- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that disagreement among predictors built on different evidence sources is a problem of evidence provenance: which source should govern a decision changes with context under non-stationarity. EPR keeps K source-specific predictors as "decision anchors". In FinBen these are local/sample, same-day contextual, temporal and relational anchors. EPR assigns one anchor to each routing context: a trading date in FinBen, a case in the legal task. Routing has three stages:
- Route initialization. A dev-fitted selector produces a base route, followed by a "relation-safety gate" that uses a frozen-LLM readout.
- Observable State–Decision Alignment (SDA). SDA proposes a switch when a candidate anchor's aggregate positive-action rate q_u(a) is closer to observable market-state summaries z_u (direction and breadth), by a normalized margin of at least τ_SDA (Eq. 3).
- Disagreement Representation & Assessment. Each proposal is serialized as a key-value "conflict packet" and encoded by frozen Qwen2.5-7B-Instruct. 101 MLP probes, trained only on synthetic packets with synthetic teacher labels, score the proposals. Proposals are ranked by normalized entropy (Eq. 4) and retained when leave-one-probe consensus F(p) > γ = 0.9 (Eqs. 5–6).
The selected anchor, not the LLM, makes the final prediction. The theory section (Thms 2.1–2.5) covers the value of context-dependent authority, the information dominance of provenance-preserving states, a three-term risk decomposition, and an assessment-regret bound. On three FinBen stock-movement datasets (ACL18, BigData22, CIKM18), EPR reaches a macro MCC of 0.196, against 0.131 for SLOT and 0.130 for TradExpert (Table 3). On ClassActionPrediction it reaches MCC 0.437, against 0.406 for LegalBERT (Table 8).
The design principle of keeping the final decision with a calibrated specialist, and using the LLM only as a representation for arbitrating, is sensible and well articulated.
Critical strengths
S1. A relevant problem framing. Separating who decides from what is predicted, and keeping domain-calibrated predictors in charge of the final output, is a principled response to the known failure modes of letting an LLM decide directly. The negative diagnostic, that outcome-trained source selectors fail to transfer chronologically (Table 4; development AUC 0.750 vs. near-chance at test), is useful if confirmed out of sample.
S2. Broad baseline families and chronological splits. The comparison covers prediction-centric models, direct LMs, LLM decision systems, decision-focused learning and adapted routers, with three-seed statistics for the trainable routers (Table 14). The authors also attempt a second domain.
S3. Extensive diagnostics and candid case analysis. The paper reports stage-wise ablations (Table 4), probe-ensemble, backbone and threshold sweeps (Tables 6, 18, 19), temporal block bootstrap (Table 7), and representative cases. The cases include a beneficial switch that was rejected (Table 5, CIKM18), which is good practice.
S4. Correct and clearly written proofs. The results in App. E are stated precisely and proved cleanly. The finite-consensus coefficients in Thm 2.5 are shown to be tight.
Critical weaknesses
W1. The evidence base is extremely thin. Routing operates at the context level: one anchor per trading date. In the final system:
- SDA emits 14 proposals across the three FinBen test sets, 10 beneficial and 4 harmful (Table 19, τ_SDA = 0.5).
- Assessment retains 8, all beneficial, for a net gain of +134 correct predictions (Table 18).
The entire +0.048 macro-MCC gain of full EPR over the gated route comes from these 8 days, and SDA's +0.033 from 14. In the legal transfer there are 6 SDA switches (4 beneficial, 2 harmful) and 5 after assessment (4/1) on 307 test cases, moving MCC from 0.417 to 0.430 to 0.437 (Table 22). Labels within a day are strongly correlated through the market factor: in Table 5, 75 of 76 samples on one day move the same way. The effective sample size is therefore closer to the number of test days than to the number of samples. There is no significance test against any baseline. The block bootstrap compares only Full against Gate or SDA, and gives Pr(Full > SDA) = 85.7–91.9% (Table 7), which falls short of conventional significance.
W2. Choices may have been selected on test outcomes. Several reported settings coincide with test-set optima:
- τ_SDA = 0.5 is the best value in the Table 19 sweep;
- the retained-proposal count n_ret = 8 is the peak in Table 6;
- Qwen2.5-7B is the best of four backbones (Table 18);
- the "candidate-action entropy (high)" readout was chosen from alternatives whose quality is shown by test harmful rates (Table 16; mean entropy over all heads gives 42.9% harmful).
W3. The decision-time claim conflicts with transductive assessment, and look-ahead is possible.
- App. B.6 (l. 920–921) says: "emitted proposals are ranked jointly within the fixed unlabeled evaluation proposal pool", with a pool retention fraction ρ = 0.5 (Table 10). Whether a switch on day t is accepted therefore depends on proposals from later test days. This is not label leakage, but it is not causal and cannot be deployed online. It is inconsistent with "evaluates authority using context-level evidence available at decision time" (Abstract, l. 20–21) and with the chronological-evaluation framing.
- Z_u "summarizes contemporaneous market direction and cross-sectional breadth" (l. 171), and the same-day anchor is built from "contemporaneous unlabeled batch summaries" (Alg. 1, l. 838). The paper must state exactly which price fields and dates these summaries use. If they include any target-day movement, they leak the label through the market factor. If they use day d−1, they should be called lagged. Because context-level MCC is dominated by market direction, this detail decides whether the main results stand.
W4. The theory is generic and does not explain the design.
- Thm 2.1 is min E ≥ E min, i.e. the value of information.
- Thm 2.2 says that a coarser σ-field cannot lower Bayes risk (Blackwell garbling).
- Cor. 2.3 follows from min(x, y) = (x + y − |x − y|)/2.
- Thm 2.4 is an add-and-subtract decomposition. Thm 2.5 combines the exact regret of a threshold rule with indicator bounds.
Questions for authors
- Z_u and the same-day anchor. For FinBen, which exact fields and dates enter z_u (direction, breadth) and the same-day anchor? Does anything derive from the target day's price movement?
- Causality. Can assessment be run online, ranking each day's proposal only against proposals from earlier days? What are the results then?
- Number of decisions. How many test days does each dataset have? How many proposals and retained switches fall in each dataset, and on which dates?
- Significance. Please give day-level paired tests of EPR against SLOT and TradExpert, and against majority-vote and stacking ensembles of your four anchors.
- Dev-split sweeps. Please show the Table 6/16/18/19 sweeps on the development split, and the process that selected τ_SDA, ρ, η, γ, the backbone and the readout head before looking at test.
- Development AUC. Is the development-set source-selection AUC of 0.750 (l. 53–64) in-sample, i.e. trained and evaluated on the development data? If so, it is not a fair contrast with the test AUC of 0.512.
Summary of the Revision
We thank all three reviewers for their careful reading and constructive feedback. We have posted detailed responses addressing each concern and question, and updated the revised manuscript with all changes highlighted in blue. We summarize the main updates below: