acceptodds
Under review as a conference paper at ICLR 2027

Evidence Drift: Evaluating Conditional Reliability in Generative Affective Dialogue Understanding

Abstract

In generative affective dialogue understanding, a correct emotion label can accompany an incorrect sentiment label, cause judgment, or support strategy. Marginal field scores and schema checks do not isolate this conditional failure. We call it evidence drift and quantify it with the conditional evidence drift rate (CEDR). We study paired prediction on MELD, RECCON, and ESConv using Qwen2.5-7B-Instruct. Under supervised joint generation, CEDR is 3.1%, 13.3%, and 36.2% for sentiment, cause judgments, and support strategies, respectively. On ESConv, this risk coexists with a strategy macro-F1 only 0.45 percentage points below that of an evidence-only expert. ESConv CEDR is 73.4% under label-constrained zero-shot generation. The failure is also observed with Mistral-7B-Instruct-v0.3 on RECCON and ESConv. We introduce Conditional Evidence Verification and Deferral (CEVD), which scores the original evidence given the dialogue and predicted emotion. Using a threshold selected on development data, it withholds evidence with weak support while preserving the primary prediction. On the held-out ESConv test set with Mistral, CEVD retains 89.3% evidence coverage and lowers CEDR among retained predictions from 36.4% to 32.6%. It outperforms random deferral at the same coverage (p = 0.0002). These findings support reporting conditional evidence error alongside marginal field performance and evidence coverage.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
affective dialogue understanding, structured generation, conditional reliability, selective prediction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_28YFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies evidence drift in generative affective dialogue understanding: a structured prediction can have a correct primary affective label (e.g., emotion) while its associated evidence field (sentiment, cause judgment, or support strategy) is incorrect. The authors define evidence drift rate (EDR) and conditional evidence drift rate (CEDR), where CEDR measures evidence error within the subset of instances whose primary prediction is correct. They evaluate on MELD (emotion–sentiment), RECCON (emotion–cause), and ESConv (emotion–strategy) using Qwen2.5-7B-Instruct and, for some settings, Mistral-7B-Instruct-v0.3. Under supervised joint generation, CEDR is 3.1%, 13.3%, and 36.2%, respectively. The paper also proposes Conditional Evidence Verification and Deferral (CEVD), which scores the original evidence conditioned on the dialogue and predicted primary label, and withholds evidence with weak support while keeping the primary prediction unchanged. On held-out Mistral–ESConv, CEVD retains 89.3% evidence coverage and lowers sCEDR from 36.4% to 32.6%, outperforming matched random deferral.

Critical strengths

The evidence-drift framing is useful and practically relevant. It isolates a failure mode that marginal field scores and schema checks do not capture: a schema-valid, primary-correct output can still carry incorrect evidence. This is a meaningful evaluation contribution for affective dialogue understanding and structured generation more broadly.

The paper is transparent about scope and limitations. The authors clearly state that cross-dataset CEDR differences are descriptive, that CEVD is a focused proof of concept, that bootstrap intervals do not model dialogue dependence, and that results measure agreement with benchmark annotations rather than clinical safety. This honesty increases trust in the reported claims.

Critical weaknesses

CEVD is validated only on ESConv with Mistral and a single verifier. The main selective-deferral result is limited to one dataset, one backbone, and one verifier architecture. The improvement is also modest: sCEDR drops from 36.4% to 32.6% at 89.3% coverage. This is a useful proof of concept, but it is not yet strong evidence that CEVD is a broadly effective reliability mechanism.

CEDR’s added value over existing joint metrics needs sharper justification. The paper notes that CEDR = 1 − JEM/Accm when primary accuracy is positive. This means CEDR is a deterministic transformation of joint exact match and primary accuracy. The metric is interpretable, but the paper should more explicitly argue what CEDR reveals beyond reporting JEM and primary accuracy together, and why it should be adopted as a separate reporting target.

Questions for authors

Can the authors extend CEVD to RECCON and MELD, and to Qwen2.5-7B-Instruct, to test whether the selective-deferral benefit generalizes beyond Mistral–ESConv?

Can the authors compare CEVD with stronger selective baselines at exactly matched coverage, such as evidence-only expert confidence, entropy, self-consistency, and a simple margin-based deferral rule?

Since CEDR is algebraically tied to JEM and primary accuracy, can the authors clarify a concrete evaluation scenario where CEDR changes a conclusion that would not be apparent from reporting JEM and primary accuracy separately?

The verifier is trained with gold primary labels but receives predicted primary labels at inference. Can the authors quantify how much CEVD’s performance depends on this mismatch, and whether the verifier is learning evidence correctness rather than exploiting primary-label information?

2 comments
AuthorsOpenReview ↗

Response to 28YF: evaluation value, selective baselines, and verifier conditioning

Thank you for recognizing the evaluation's practical value and transparent scope. We address your four questions below, separating reported results from mathematical clarification.

1. Scope and magnitude of the CEVD result.

The diagnostic study covers three datasets and two model families; the CEVD intervention is evaluated only on Mistral-ESConv with one verifier. We do not infer cross-dataset or cross-backbone effectiveness from the diagnostic results.

Table 3 and Appendix H report two development-selected operating points. Rates are percentages; Capture is the fraction of original drift cases withheld.

SettingTest coveragesCEDRCapture
No deferral100.0036.37—
90% development target89.3332.5719.95
80% development target79.0530.3333.58

The reductions are 3.80 and 6.04 percentage points at reduced evidence coverage. Primary predictions and retained evidence labels remain unchanged. These existing results demonstrate targeted selective release in one setting, not broad method generalization. CEVD on Qwen and on MELD/RECCON remains untested. We maintain the distinction between the multi-setting diagnostic contribution and this mitigation proof of concept.

2. Stronger baselines at exactly matched coverage.

The submitted comparison with random deferral matches the number of retained predictions exactly. Its empirical p=0.0002 concerns that control only. Appendix H also reports entropy and disagreement, but their realized coverages differ. For example, at the 90% development target, coverage/sCEDR are 89.33/32.57 for support, 88.78/33.20 for entropy, and 89.83/32.97 for disagreement. These results do not establish superiority at matched coverage.

The requested expert-confidence, margin, and sampled self-consistency comparisons are not reported. Hybrid-Compose replaces evidence labels and is not an expert-confidence deferral baseline; the reported disagreement score is not a substitute for sampled self-consistency. A fair comparison must keep the original predictions fixed, score their original evidence, retain the same number under the same eligibility rule, and report both PCov and sCEDR. Offline ranking must be distinguished from development-threshold transfer. Our superiority claim is limited to matched random deferral.

3. What CEDR adds to reporting JEM and primary accuracy.

We agree that CEDR adds no independent information when JEM and primary accuracy are both available. Its purpose is to state the conditional evaluation target explicitly, not to introduce a new probability identity.

Consider an arithmetic illustration, not an experimental result, with 1,000 instances per system:

SystemPrimary correctJoint correctPrimary accuracy (%)JEM (%)CEDR (%)
A80072080.072.010.0
B90076590.076.515.0

Both primary accuracy and JEM improve in B, but evidence error conditional on primary correctness increases from 10% to 15%. B is better for full-pair success, while A has lower conditional error. This distinction is fully recoverable from the original two metrics; explicitly reporting CEDR makes the objective tradeoff visible. It is not a reason to rank systems by CEDR alone, because their conditioning subsets can differ. Section 3.2 therefore reports it with primary accuracy and conditioning-set size.

4. Gold-primary training versus predicted-primary inference.

For a fixed verifier, define S as the instances where the generated primary label is correct. For every eligible instance in S, the following equality holds:

pϕ(e^i∣Xi,m^i)=pϕ(e^i∣Xi,mi)p_\phi(\hat e_i\mid X_i,\hat m_i)=p_\phi(\hat e_i\mid X_i,m_i).

Thus, substituting gold for predicted primary labels has zero direct score effect on this subset; with a fixed threshold and eligibility gate, retention is identical. This is an algebraic statement, not a measured ablation. Test references identify S only for evaluation, not for release decisions.

It does not make the overall mismatch irrelevant. Primary errors can affect development risks and hence the selected threshold, overall coverage, and behavior outside S. Nor does it imply that training on predicted labels would learn the same verifier.

Equation (5) learns a conditional evidence-label distribution, not a binary correctness classifier. Primary-label information is intentionally used; what remains unisolated is the contribution of dialogue beyond label priors. Context-only and primary-only controls, alongside training with out-of-fold predicted primary labels, would distinguish these effects without using test labels. We do not equate selective error reduction with proof that the verifier avoids label-prior shortcuts.

AuthorsOpenReview ↗

Follow-up: summary of clarifications and remaining questions

Thank you again for your constructive review and for recognizing the value of the conditional-reliability perspective. We would appreciate your feedback on our response above. For convenience, we summarize the clarifications corresponding to your four questions.

1. Scope of the CEVD evaluation. The evidence-drift analysis spans three datasets and two model families, whereas CEVD is evaluated as a proof of concept on Mistral–ESConv with one verifier. We distinguish these scopes rather than treating the broader diagnostic results as evidence that CEVD generalizes across all settings.

2. Selective baselines. The random-deferral comparison matches the retained count exactly. The reported entropy and disagreement comparisons have different realized coverages, so they do not establish superiority at exactly matched coverage. We also distinguish evidence replacement by Hybrid-Compose from using expert confidence to select the original evidence predictions.

3. Interpretation of CEDR. We agree that CEDR contains no independent information beyond primary accuracy and JEM. Its value is to make the conditional evaluation target explicit. The arithmetic example in our response illustrates how primary accuracy and JEM can both improve while conditional evidence error increases. We therefore advocate reporting CEDR alongside those metrics and the conditioning-set size, not replacing them.

4. Verifier conditioning. For a fixed verifier, threshold, and eligibility rule, replacing predicted primary labels with gold labels leaves scores and retention decisions unchanged on primary-correct instances, where those labels already coincide. This pointwise equality does not eliminate possible effects on development-set threshold selection, overall coverage, or learning during training, nor does it establish that the verifier avoids reliance on label priors.

Could you let us know whether these clarifications address your questions about the evaluation objective and verifier conditioning, and which concerns remain most important for your assessment? Please also let us know if there are additional points that would benefit from clarification.

Thank you for your time and consideration.

Reviewer_bfnXOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the evidence drift phenomeny in generative affective dialogue understanding task where a correct emotion label can accompany wrong sentiment labels, cause judgment or supply strategy. The authors conducted experiments on three benchmark for two models, i.e., Qwen2.5 7B and Mistral 7B. They observed that both LLMs did not outperforms an evidence-only expert. Thus, the CEVD method is designed by this work, working with selected threshold to filter out unnecessary evidence and thus improve the model performance.

Critical strengths
  1. An interesting findings on the generative affective dialogue understanding task
  2. Many experiments and analysis were conducted
Critical weaknesses
  1. The proposed method is tricky since the selected threshold cannot generalize well potentially.
  2. The used LLM models are outdated which makes the findings unconvincable.
  3. The baseline methods are very simple, like random deferral.
Questions for authors

Refer to weakness section

2 comments
AuthorsOpenReview ↗

Response to bfnX: threshold sensitivity and synthetic selector controls

Thank you for the comments. CEVD reduces error among released evidence by withholding some evidence fields; it does not correct labels or improve full-coverage predictions. We clarify the original protocol and supplement it with explicitly synthetic diagnostics.

1. Threshold selection and generalization.

Section 3.3 selects a risk quantile on development data and applies that numerical threshold unchanged to test data. The targets 0.80/0.90 are coverage levels, not confidence thresholds. The reported Mistral-ESConv test coverages are 79.05%/89.33%, with sCEDR 30.33%/32.57% versus 36.37% without deferral. These are existing results. They establish neither calibrated correctness probabilities nor robustness to distribution shift.

To examine threshold sensitivity separately, we fitted small classifiers on artificial features with imbalanced labels, unequal dialogue lengths, and shared dialogue difficulty. Each world/seed uses 6,000 model-training, 2,000 additional error-selector-training and 1,600 development records, with 2,400 records per test scenario. No ESConv dialogue, LLM, or DeBERTa inference is involved. The tables below use the first prespecified seed (17) with the assumed primary-evidence relation present; the archive retains seeds 17/29/43 and both relation-present/removed worlds.

Table R1. Synthetic development-set sensitivity.

Models and test predictions are fixed. We resample development dialogues 1,000 times and recompute thresholds. Entries are medians [2.5th, 97.5th percentiles]; coverage and sCEDR are percentages.

TargetRisk thresholdTest coveragesCEDR
80%0.6865 [0.6215, 0.7392]81.62 [79.08, 83.71]17.31 [15.73, 18.40]
90%0.8962 [0.8625, 0.9331]91.12 [89.42, 92.96]22.99 [21.89, 23.85]

These ranges describe development-sample variability, not training uncertainty or a risk guarantee.

Table R2. Synthetic fixed-threshold stress test.

The same source-development threshold, τ90=0.8962\tau_{90}=0.8962, is used throughout. Rates are percentages.

Artificial test scenarioNo-deferral CEDRTest coveragesCEDR
Same generating mechanism27.7691.1222.99
Difficulty/group-mixture shift36.8189.0830.35
Primary-evidence relation shift44.7187.1241.80
Support-scale change only27.7694.8825.04

The last row reuses the first row’s labels and predictions and applies a stipulated monotone support transformation. Ranking is unchanged, but the fixed threshold retains more predictions. This illustrates the distinction between ranking quality and threshold transfer, not robustness of the actual CEVD verifier. We therefore retain the manuscript’s explicit restriction to empirical coverage-error tradeoffs.

2. Model coverage.

The original evidence concerns Qwen2.5-7B on three datasets and Mistral-7B on RECCON/ESConv, not current LLMs generally. The simulations above do not address model age or replace newer-model evaluation. That concern remains open; our empirical claims are limited to the tested models and settings.

3. Baseline strength.

Random deferral tests whether selection is better than discarding the same number indiscriminately; it is not a strong learned-selector baseline. Section 4.4 and Appendix H also report entropy and disagreement, but their realized coverages differ and their AUROCs are close. We do not claim matched-coverage superiority from those results.

Table R3. Synthetic selector comparison.

Here, every method retains exactly 2,160/2,400 test records, using scores without test labels and a shared legality gate. This is offline ranking, not development-threshold transfer. PCov is coverage within the primary-correct subset; rates are percentages. All method names below denote simulated selectors.

Simulated selectorPCovsCEDR
Conditional support91.9722.26
Unconditional support91.3722.03
Predictor support90.9823.16
Entropy91.6724.01
Probability margin91.3724.25
Disagreement gap91.7722.15
Learned error selector91.2721.94
Random within legal pool90.4727.22

Conditional/unconditional verifiers use the same logistic family and training set but different input dimensions; the learned selector uses 2,000 additional labeled records. Random results average 2,000 trials. The conditional-minus-unconditional sCEDR difference is +0.23 pp, with a paired dialogue-bootstrap interval of [−0.71, +1.15] pp. Under the artificial relation shift it is +1.96 [+1.15, +2.79] pp (2,000 resamples each; fitted models and selection masks fixed). These results do not establish conditional superiority. They delimit the synthetic analysis and motivate keeping it separate from claims about the original neural method.

AuthorsOpenReview ↗

Follow-up: summary of clarifications and remaining questions

Thank you again for your review. We would appreciate your feedback on our response above. For convenience, we briefly summarize the clarifications corresponding to your three concerns.

1. Threshold selection. The numerical threshold is selected from development-set risks and applied unchanged to the held-out test set, rather than optimized for test performance. We report realized coverage and selective error at both operating points. This supports an empirical coverage–error tradeoff in the evaluated setting, not a guarantee under distribution shift.

2. Model coverage. Our model-based findings are limited to the evaluated Qwen2.5-7B and Mistral-7B configurations, with CEVD assessed as a proof of concept on Mistral–ESConv. The supplementary simulations examine sensitivity under specified assumptions; they are separate from, and do not replace, evaluations of newer models.

3. Baseline comparisons. Random deferral controls for the effect of withholding the same number of predictions indiscriminately. The manuscript also reports entropy and disagreement scores, but their realized coverages differ, so we do not claim superiority over these alternatives at exactly matched coverage. CEVD preserves the primary predictions and retained evidence labels; its demonstrated benefit is lower error among selectively released evidence.

Could you let us know which of your concerns remain unresolved after these clarifications, or whether there are additional points that need explanation? In particular, understanding which issue is most consequential for your assessment would help us focus the discussion.

Thank you for your time and consideration.

Reviewer_YSJTOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper defines "evidence drift": in two-field affective prediction (emotion + sentiment / cause / strategy), the primary emotion is correct but the associated field is wrong. It quantifies this with CEDR, i.e., P(evidence wrong | primary correct), on MELD, RECCON and ESConv using Qwen2.5-7B-Instruct, with Mistral-7B-Instruct-v0.3 as a second backbone. It also proposes CEVD, which uses a DeBERTaV3 verifier to withhold low-support evidence while keeping the primary prediction. On Mistral–ESConv, CEVD reduces CEDR from 36.4% to 32.6% at 89.3% coverage.

Critical strengths

None

Critical weaknesses

1. The problem is trivial. CEDR is just a conditional error rate; the paper itself states CEDR = 1 − JEM/Acc_m. Giving a standard quantity a new name ("evidence drift") is not a conceptual contribution.

2. The phenomenon is likely an artifact of weak models and a fixed output order.

  • Only two 7B models from 2023–2024 are tested, each fine-tuned for one epoch. Stronger instruction-following models may show much less mismatch, so it is unclear whether evidence drift is a general property of LLMs or a limitation of these specific models.
  • The JSON schema always places the primary field before the evidence field (Appendix B.1). The model therefore commits to an emotion before considering the cause or strategy.
  • Reasoning ("thinking") models, or simply reversing the field order so the model produces evidence first, could plausibly reduce the problem.

3. Presentation. Figures 1–2 are cluttered and hard to read: many icons and boxes, little information.

Questions for authors
  1. Stronger models and reasoning. Does evidence drift persist with stronger, more recent instruction-following models? Does it persist with reasoning ("thinking") models that produce a rationale before the labels? Even a zero-shot evaluation on the ESConv test set would help show whether the phenomenon is general.

  2. Output order. What happens to CEDR if the JSON schema puts the evidence field before the primary field, so the model predicts cause or strategy first? If the order changes CEDR substantially, the "drift" may be largely a decoding-order artifact.

3 comments
AuthorsOpenReview ↗

Response to YSJT: contribution and additional synthetic diagnostics

Thank you for the review. We have supplemented the analysis with synthetic simulation experiments, reported below separately from the submitted model results. These experiments examine the evaluation procedure; they do not constitute tests of stronger LLMs.

1. Contribution beyond the CEDR identity. Section 3.2 explicitly states CEDR = 1 - JEM/Acc_m. We do not claim a new probability identity. The contribution is the empirical analysis of evidence reliability after a correct primary prediction and the selective release of evidence.

The submitted Hybrid-Compose control preserves Joint-LoRA's primary predictions and replaces only its evidence with an independently generated expert prediction. On the same 1,152 primary-correct ESConv instances, CEDR is 36.20% for Joint-LoRA and 36.46% for Hybrid-Compose, despite the expert's higher marginal strategy F1 (58.41% versus 57.96%). This point-estimate comparison shows why better marginal performance does not necessarily imply better conditional performance; we do not claim a significant difference. Separately, CEVD retains 89.33% of evidence while reducing sCEDR from 36.37% to 32.57%, outperforming matched random deferral (Table 3). These are existing results, not newly conducted model experiments.

2. Additional synthetic simulation experiments. We generated 1,809 artificial records grouped into 150 simulated dialogues, with unequal dialogue lengths, imbalanced labels, shared dialogue difficulty, instance difficulty, and structural failures. Capability, reasoning, and order effects are hypothetical simulation parameters, not measurements of neural models. No real ESConv dialogue or Qwen/Mistral inference was used.

Table R1 reports the first prespecified generation seed (17); results for seeds 29 and 43 are also retained in the simulation outputs. L0, H0, and H1 denote SIM_LOWER_R0, SIM_HIGHER_R0, and SIM_HIGHER_R1. R0/R1 disable/enable an assumed reasoning-related parameter shift, not actual chain-of-thought inference. Here, m is the primary field and e the evidence field; rates are percentages.

Table R1. Additional synthetic experiments, not LLM benchmarks.

ProfileOrderPrimary acc.Evidence F1Drift/primary-correctCEDR
L0m→e73.0258.70434/1,32132.85
L0e→m69.7663.31346/1,26227.42
H0m→e79.3368.50349/1,43524.32
H0e→m77.1773.24284/1,39620.34
H1m→e81.4870.62350/1,47423.74
H1e→m78.3975.60256/1,41818.05

3. Paired analysis of the simulated orders. Because the primary-correct subset can change with order, Table R2 compares evidence errors on instances where the primary prediction is correct under both settings. Differences are reverse minus forward, in percentage points (pp). Intervals use 2,000 paired bootstrap resamples of simulated dialogues. Corrected/harmed counts describe evidence errors removed/introduced by the reverse setting on this common subset.

Table R2. Shared-subset comparison, all results synthetic.

ProfileCommon nForward error (%)Reverse error (%)Difference (pp)95% interval (pp)Corrected/harmed
L01,23831.4227.71-3.72[-5.18, -2.20]68/22
H01,37223.6220.19-3.43[-4.71, -2.22]60/13
H11,39122.7218.26-4.46[-5.62, -3.32]71/9

For L0, the own-subset CEDR reduction is 5.44 pp, whereas the common-subset reduction is 3.72 pp, calculated before rounding. This illustrates why an order comparison should distinguish changes in evidence predictions from changes in the conditioning set. The direction and magnitude depend on the assumed generator; these intervals are not evidence of real-model improvements.

4. Scope and presentation. The simulation supplements the diagnostic analysis but cannot resolve the requested stronger-model, reasoning-model, or actual decoding-order controls. Those remain untested. Likewise, independently generated evidence in Hybrid-Compose provides a related control but does not exclude an order effect in joint generation. We restrict our model conclusions to the evaluated configurations and do not claim convergence after one epoch.

We accept the figure-readability concern. We propose simplifying Figure 1 to a reference/prediction comparison and Figure 2 to separate offline evaluation from CEVD inference, removing decorative elements and enlarging labels.

Thank you for your detailed response. However, my main concerns remain unresolved. In particular, the synthetic simulations do not substitute for empirical evaluations on stronger reasoning models or actual decoding-order ablations. These experiments are essential to establish whether the reported phenomenon is generalizable rather than an artifact of limited model capability or output ordering. I also remain unconvinced by the methodological novelty. Therefore, I will maintain my original score.

AuthorsOpenReview ↗

Follow-up to Reviewer YSJT on empirical scope and contribution

Thank you for identifying the remaining concerns. We ran GPT-6.2 Sol, Claude Sonnet 5.5, and Qwen/Qwen3.5-27B on the same 1,809 synthetic evaluation records, grouped into 603 dialogue turns. We computed CEDR from the models’ returned predictions. For each model, we compared two requested output orders during generation under two inference settings. A and B share the first setting, and C and D share the second. A and C request emotion before strategy. B and D request strategy before emotion.

Supplementary results. The CEDR columns use each condition’s own primary-correct subset. Each paired comparison uses only items whose primary label was predicted correctly under both orders. Paired differences are strategy-first minus emotion-first in percentage points (pp). Brackets give the reported pointwise 95% intervals from 5,000 dialogue-level paired bootstrap resamples.

ModelA CEDR (%)B CEDR (%)C CEDR (%)D CEDR (%)Paired A → B (pp)Paired C → D (pp)
GPT-6.2 Sol36.3438.6834.8432.34+4.04 [2.33, 5.92]−1.73 [−3.25, −0.08]
Claude Sonnet 5.533.2834.7035.5433.58+1.56 [0.00, 3.15]−1.13 [−2.72, 0.52]
Qwen3.5-27B44.0343.4345.0442.10−0.19 [−2.08, 1.67]−3.30 [−5.07, −1.54]

Conditional evidence errors persisted across all 12 model–condition combinations. The paired results also show that the requested output order can affect evidence reliability in either direction. For GPT-6.2 Sol under A → B, strategy-first corrected 28 evidence errors and introduced 76 among 1,187 shared primary-correct cases (+4.04 pp). For Qwen3.5-27B under C → D, it corrected 65 and introduced 30 among 1,061 such cases (−3.30 pp). These results extend the evaluation to three additional LLMs and both prompted orders on a common synthetic test set. The magnitude and direction of order effects on natural dialogues remain open.

Contribution of the submitted study. We agree that CEDR is determined by JEM and primary accuracy through the identity in Section 3.2. The contribution is empirical. First, Hybrid-Compose holds primary predictions fixed. On the same 1,152 primary-correct ESConv instances, the evidence-only expert had higher full-test strategy F1 (58.41% vs. 57.96%), while CEDR was 36.46% after replacement and 36.20% for Joint-LoRA. These point estimates show that the two evaluations can rank the same evidence predictions differently. Second, on Mistral-ESConv, CEVD reduced sCEDR from 36.37% to 32.57% at a reported evidence retention of 89.33%, with lower sCEDR than random deferral at the same retained count (Table 3). Together, these comparisons show why it matters to measure evidence reliability after a correct primary prediction and to evaluate selective release at a matched retained count.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.