acceptodds
Under review as a conference paper at ICLR 2027

Evidence Provenance Routing for Robust Decision Making under Predictor Disagreement

24 views

Abstract

Modern decision pipelines increasingly rely on heterogeneous predictors trained over distinct evidence sources. While these models often achieve comparable aggregate accuracy, they frequently produce conflicting predictions on individual instances—leading to brittle downstream decisions under non-stationary conditions. We show that a fundamental driver of this disagreement is evidence provenance: different predictors observe distinct evidence scopes whose relevance shifts dynamically across environments. To address this, we introduce Evidence Provenance Routing (EPR), a framework that maintains source-specific predictors as explicit decision anchors and dynamically allocates source authority during prediction conflicts. Unlike prior routing methods relying on static confidence metrics or historical accuracy, EPR evaluates authority using context-level evidence available at decision time. Specifically, EPR decouples routing into two stages: (1) Observable State–Decision Alignment (SDA), which identifies candidate source switches based on observable context, and (2) Disagreement Representation & Assessment, where a frozen language model encodes conflict semantics and lightweight probes validate stable route updates. Crucially, the final decision is executed by the selected source anchor, preserving domain-specific calibration rather than delegating task generation to the LLM. Evaluated across three non-stationary financial decision benchmarks and a legal outcome-prediction dataset, EPR consistently outperforms prediction aggregation, direct LLM reasoning, decision-focused learning, and modern router baselines. Finally, we provide a decision-theoretic analysis characterizing the value of context-dependent authority, bounding the decision cost of provenance compression, and decomposing routing risk across information, proposal and assessment stages.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
robust decision making, decision routing, predictor disagreement, heterogeneous evidence, evidence provenance, model routing, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_5FWgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that disagreement among predictors built on different evidence sources is a problem of evidence provenance: which source should govern a decision changes with context under non-stationarity. EPR keeps K source-specific predictors as "decision anchors". In FinBen these are local/sample, same-day contextual, temporal and relational anchors. EPR assigns one anchor to each routing context: a trading date in FinBen, a case in the legal task. Routing has three stages:

  1. Route initialization. A dev-fitted selector produces a base route, followed by a "relation-safety gate" that uses a frozen-LLM readout.
  2. Observable State–Decision Alignment (SDA). SDA proposes a switch when a candidate anchor's aggregate positive-action rate q_u(a) is closer to observable market-state summaries z_u (direction and breadth), by a normalized margin of at least τ_SDA (Eq. 3).
  3. Disagreement Representation & Assessment. Each proposal is serialized as a key-value "conflict packet" and encoded by frozen Qwen2.5-7B-Instruct. 101 MLP probes, trained only on synthetic packets with synthetic teacher labels, score the proposals. Proposals are ranked by normalized entropy (Eq. 4) and retained when leave-one-probe consensus F(p) > γ = 0.9 (Eqs. 5–6).

The selected anchor, not the LLM, makes the final prediction. The theory section (Thms 2.1–2.5) covers the value of context-dependent authority, the information dominance of provenance-preserving states, a three-term risk decomposition, and an assessment-regret bound. On three FinBen stock-movement datasets (ACL18, BigData22, CIKM18), EPR reaches a macro MCC of 0.196, against 0.131 for SLOT and 0.130 for TradExpert (Table 3). On ClassActionPrediction it reaches MCC 0.437, against 0.406 for LegalBERT (Table 8).

The design principle of keeping the final decision with a calibrated specialist, and using the LLM only as a representation for arbitrating, is sensible and well articulated.

Critical strengths

S1. A relevant problem framing. Separating who decides from what is predicted, and keeping domain-calibrated predictors in charge of the final output, is a principled response to the known failure modes of letting an LLM decide directly. The negative diagnostic, that outcome-trained source selectors fail to transfer chronologically (Table 4; development AUC 0.750 vs. near-chance at test), is useful if confirmed out of sample.

S2. Broad baseline families and chronological splits. The comparison covers prediction-centric models, direct LMs, LLM decision systems, decision-focused learning and adapted routers, with three-seed statistics for the trainable routers (Table 14). The authors also attempt a second domain.

S3. Extensive diagnostics and candid case analysis. The paper reports stage-wise ablations (Table 4), probe-ensemble, backbone and threshold sweeps (Tables 6, 18, 19), temporal block bootstrap (Table 7), and representative cases. The cases include a beneficial switch that was rejected (Table 5, CIKM18), which is good practice.

S4. Correct and clearly written proofs. The results in App. E are stated precisely and proved cleanly. The finite-consensus coefficients in Thm 2.5 are shown to be tight.

Critical weaknesses

W1. The evidence base is extremely thin. Routing operates at the context level: one anchor per trading date. In the final system:

  • SDA emits 14 proposals across the three FinBen test sets, 10 beneficial and 4 harmful (Table 19, τ_SDA = 0.5).
  • Assessment retains 8, all beneficial, for a net gain of +134 correct predictions (Table 18).

The entire +0.048 macro-MCC gain of full EPR over the gated route comes from these 8 days, and SDA's +0.033 from 14. In the legal transfer there are 6 SDA switches (4 beneficial, 2 harmful) and 5 after assessment (4/1) on 307 test cases, moving MCC from 0.417 to 0.430 to 0.437 (Table 22). Labels within a day are strongly correlated through the market factor: in Table 5, 75 of 76 samples on one day move the same way. The effective sample size is therefore closer to the number of test days than to the number of samples. There is no significance test against any baseline. The block bootstrap compares only Full against Gate or SDA, and gives Pr(Full > SDA) = 85.7–91.9% (Table 7), which falls short of conventional significance.

W2. Choices may have been selected on test outcomes. Several reported settings coincide with test-set optima:

  • τ_SDA = 0.5 is the best value in the Table 19 sweep;
  • the retained-proposal count n_ret = 8 is the peak in Table 6;
  • Qwen2.5-7B is the best of four backbones (Table 18);
  • the "candidate-action entropy (high)" readout was chosen from alternatives whose quality is shown by test harmful rates (Table 16; mean entropy over all heads gives 42.9% harmful).

W3. The decision-time claim conflicts with transductive assessment, and look-ahead is possible.

  • App. B.6 (l. 920–921) says: "emitted proposals are ranked jointly within the fixed unlabeled evaluation proposal pool", with a pool retention fraction ρ = 0.5 (Table 10). Whether a switch on day t is accepted therefore depends on proposals from later test days. This is not label leakage, but it is not causal and cannot be deployed online. It is inconsistent with "evaluates authority using context-level evidence available at decision time" (Abstract, l. 20–21) and with the chronological-evaluation framing.
  • Z_u "summarizes contemporaneous market direction and cross-sectional breadth" (l. 171), and the same-day anchor is built from "contemporaneous unlabeled batch summaries" (Alg. 1, l. 838). The paper must state exactly which price fields and dates these summaries use. If they include any target-day movement, they leak the label through the market factor. If they use day d−1, they should be called lagged. Because context-level MCC is dominated by market direction, this detail decides whether the main results stand.

W4. The theory is generic and does not explain the design.

  • Thm 2.1 is min E ≥ E min, i.e. the value of information.
  • Thm 2.2 says that a coarser σ-field cannot lower Bayes risk (Blackwell garbling).
  • Cor. 2.3 follows from min(x, y) = (x + y − |x − y|)/2.
  • Thm 2.4 is an add-and-subtract decomposition. Thm 2.5 combines the exact regret of a threshold rule with indicator bounds.
Questions for authors
  1. Z_u and the same-day anchor. For FinBen, which exact fields and dates enter z_u (direction, breadth) and the same-day anchor? Does anything derive from the target day's price movement?
  2. Causality. Can assessment be run online, ranking each day's proposal only against proposals from earlier days? What are the results then?
  3. Number of decisions. How many test days does each dataset have? How many proposals and retained switches fall in each dataset, and on which dates?
  4. Significance. Please give day-level paired tests of EPR against SLOT and TradExpert, and against majority-vote and stacking ensembles of your four anchors.
  5. Dev-split sweeps. Please show the Table 6/16/18/19 sweeps on the development split, and the process that selected τ_SDA, ρ, η, γ, the backbone and the readout head before looking at test.
  6. Development AUC. Is the development-set source-selection AUC of 0.750 (l. 53–64) in-sample, i.e. trained and evaluated on the development data? If so, it is not a fair contrast with the test AUC of 0.512.
Reviewer_nMMEOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Evidence Provenance Routing (EPR), which selects among predictors associated with different evidence scopes. It combines initial routing, state-alignment-based switching proposals, and probe-consensus assessment using frozen LLM representations. Experiments report improvements on three financial prediction benchmarks and a legal outcome prediction task, accompanied by theoretical analysis, component ablations, and robustness checks.

Critical strengths
  1. The motivation is interesting: predictors may disagree because they rely on different evidence, and explicitly considering evidence provenance could improve decision-making.
  2. The paper provides useful robustness analyses, including component ablations, parameter sensitivity, probe and backbone comparisons, and temporal block-bootstrap evaluation.
Critical weaknesses
  1. Heavy heuristic design with limited justification. The framework combines several handcrafted scores, filtering rules, and thresholds. The theory and ablations do not sufficiently establish why these particular choices and their combination are necessary.

  2. Limited relevance of the evaluation. Small, dated offline prediction datasets do not establish usefulness in modern agentic financial decision-making. Stronger LLM baselines using simpler evidence integration or predictor selection are also needed to demonstrate the framework’s added value.

  3. The evidence-provenance motivation remains insufficiently validated. The anchors differ in both information scope and construction, making it unclear whether gains come from provenance modeling or ordinary predictor complementarity. A controlled provenance ablation would help distinguish these explanations.

Questions for authors

Questions

  1. How can the improvements be attributed specifically to evidence provenance, rather than differences in predictor architecture, feature construction, or ordinary model complementarity?
  2. Why should market-state alignment, high conflict-classification entropy, and probe-selection stability indicate beneficial predictor switches? What evidence establishes these connections beyond aggregate performance gains?
  3. Does EPR retain its advantage over a strong LLM directly integrating the same evidence or selecting among the same predictors? How dependent are the gains on the particular anchor pool?

Suggestions

Clarify how the anchors are constructed, what information each receives, and what is being routed. Streamline the formalism in Sec. 2 to make the concrete method and its connection to the motivation easier to understand.

Reviewer_hjMqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Evidence Provenance Routing (EPR) for choosing among predictors that use different evidence sources. It keeps these predictors as separate decision anchors and selects one anchor for each context. After route initialization, observable state summaries propose source switches, and probes over frozen LLM representations assess which switches to retain. The final prediction comes from the selected anchor. The paper also gives an information-based analysis of routing risk. Experiments on three chronological financial datasets and a legal outcome dataset show improvements over the reported baselines and the initial routes. The main concerns are whether the assessment protocol matches the claimed decision-time setting and whether the probe score reliably identifies beneficial switches.

Critical strengths
  • The paper addresses a useful problem: predictors based on different sources can disagree, and the best source may change with the context. Keeping the source predictors separate makes the routing decision easier to inspect, while separating proposal generation from assessment is a reasonable design.

  • The component analysis is informative. Table 4 shows gains from the relation gate, state-based proposals, and final assessment. The temporal block bootstrap and the alternative-backbone results provide more evidence than a comparison of average scores alone. The legal experiment also shows that the framework can be instantiated with different evidence sources.

Critical weaknesses
  • The connection between the evaluation protocol and decision-time routing needs clarification. Appendix B.6 ranks proposals jointly within a fixed unlabeled evaluation pool and retains them using pool-level fractions and group constraints. If this pool includes multiple test dates, an earlier decision can depend on proposals from later dates, even without using their labels. This differs from choosing a source using information available on that date. Chronological train/test splits alone do not resolve this issue. The scope of the pool and performance under a strictly chronological implementation are important to the paper's main claim.

  • The link between probe stability and useful switching is not sufficiently established. The probes learn synthetic conflict categories rather than whether a switch improves the task outcome, and proposals with higher classification entropy are preferred. Stable ranking across probe initializations does not by itself imply positive switch value. The theoretical risk decomposition does not establish this relationship either. The empirical assessment gain is encouraging, but it rests on 14 financial proposals with eight retained switches; in the legal experiment, assessment removes one harmful switch (Table 22). Appendix B.5 also does not give concrete definitions and assignment rules for the candidate-action-family labels. These limitations make it difficult to judge whether the assessment captures a transferable conflict signal or a pattern specific to this small proposal set.

Questions for authors
  • Does the evaluation proposal pool in Appendix B.6 contain proposals from later test dates, or from multiple datasets? Could the authors report a chronological version whose retention rule is fixed on development data or updated only from past proposals, without revising earlier decisions?

  • What exactly are the candidate-action-family classes and how does the synthetic teacher assign them? Why should high entropy over these classes indicate a beneficial switch? A comparison using the same synthetic labels and consensus rule on raw packet features, without the frozen LLM, would help clarify the contribution of the representation.

  • How were the assessment head, backbone, retention fraction, and group constraints selected before test evaluation? Can the authors provide the corresponding development results and assess the incremental gain over SDA on additional chronological windows or legal splits? This would help determine whether the final assessment benefit extends beyond the small set of switches reported here.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Revision

We thank all three reviewers for their careful reading and constructive feedback. We have posted detailed responses addressing each concern and question, and updated the revised manuscript with all changes highlighted in blue. We summarize the main updates below:

  • Research question and theoretical motivation — 5FWg and nMME. We clarified EPR’s central question of which evidence source should govern a decision when specialist predictors disagree. The revision connects the theory directly to three design choices: preserving source identity, separating switch proposal from commitment, and using probe consensus to audit switch stability.
  • Decision-time inputs and chronological assessment — 5FWg and hjMq. We verified across all records that price inputs strictly precede the target date and reported the exact proposal and retained-switch dates. We also evaluated a strictly chronological assessment replay using only preceding proposals within each dataset, reproducing all 14 pooled decisions and the exact same 0.1961 macro MCC.
  • Conflict representation and switch assessment — hjMq and nMME. We detailed how the synthetic teacher assigns candidate source-family labels and how the resulting readout ranks proposed switches. Under matched supervision and consensus rules, the frozen LLM’s contextual representation improves macro MCC (0.1961 vs. 0.1733) and eliminates harmful switches compared with the raw-field control.
  • Controlled comparisons with shared predictors — 5FWg and nMME. Holding the four specialist anchors fixed, we added majority voting, stacking, two frontier open-source LLMs, and three open-source decision models as baselines. Paired block-bootstrap tests confirm EPR’s statistically significant gains over all seven alternatives (alongside confidence bounds over published baselines), and EPR likewise outperforms all five model-based assessment replacements on the same proposals.
  • Configuration protocol and evaluation diagnostics — 5FWg and hjMq. We clarified how the proposal margin, retention budget, group coverage floor, and consensus threshold operate together, summarized sensitivity across thresholds, readouts, and frozen encoders, and detailed the rolling out-of-fold development AUC evaluation against matched chronological test splits.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.