- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies evidence drift in generative affective dialogue understanding: a structured prediction can have a correct primary affective label (e.g., emotion) while its associated evidence field (sentiment, cause judgment, or support strategy) is incorrect. The authors define evidence drift rate (EDR) and conditional evidence drift rate (CEDR), where CEDR measures evidence error within the subset of instances whose primary prediction is correct. They evaluate on MELD (emotion–sentiment), RECCON (emotion–cause), and ESConv (emotion–strategy) using Qwen2.5-7B-Instruct and, for some settings, Mistral-7B-Instruct-v0.3. Under supervised joint generation, CEDR is 3.1%, 13.3%, and 36.2%, respectively. The paper also proposes Conditional Evidence Verification and Deferral (CEVD), which scores the original evidence conditioned on the dialogue and predicted primary label, and withholds evidence with weak support while keeping the primary prediction unchanged. On held-out Mistral–ESConv, CEVD retains 89.3% evidence coverage and lowers sCEDR from 36.4% to 32.6%, outperforming matched random deferral.
Critical strengths
The evidence-drift framing is useful and practically relevant. It isolates a failure mode that marginal field scores and schema checks do not capture: a schema-valid, primary-correct output can still carry incorrect evidence. This is a meaningful evaluation contribution for affective dialogue understanding and structured generation more broadly.
The paper is transparent about scope and limitations. The authors clearly state that cross-dataset CEDR differences are descriptive, that CEVD is a focused proof of concept, that bootstrap intervals do not model dialogue dependence, and that results measure agreement with benchmark annotations rather than clinical safety. This honesty increases trust in the reported claims.
Critical weaknesses
CEVD is validated only on ESConv with Mistral and a single verifier. The main selective-deferral result is limited to one dataset, one backbone, and one verifier architecture. The improvement is also modest: sCEDR drops from 36.4% to 32.6% at 89.3% coverage. This is a useful proof of concept, but it is not yet strong evidence that CEVD is a broadly effective reliability mechanism.
CEDR’s added value over existing joint metrics needs sharper justification. The paper notes that CEDR = 1 − JEM/Accm when primary accuracy is positive. This means CEDR is a deterministic transformation of joint exact match and primary accuracy. The metric is interpretable, but the paper should more explicitly argue what CEDR reveals beyond reporting JEM and primary accuracy together, and why it should be adopted as a separate reporting target.
Questions for authors
Can the authors extend CEVD to RECCON and MELD, and to Qwen2.5-7B-Instruct, to test whether the selective-deferral benefit generalizes beyond Mistral–ESConv?
Can the authors compare CEVD with stronger selective baselines at exactly matched coverage, such as evidence-only expert confidence, entropy, self-consistency, and a simple margin-based deferral rule?
Since CEDR is algebraically tied to JEM and primary accuracy, can the authors clarify a concrete evaluation scenario where CEDR changes a conclusion that would not be apparent from reporting JEM and primary accuracy separately?
The verifier is trained with gold primary labels but receives predicted primary labels at inference. Can the authors quantify how much CEVD’s performance depends on this mismatch, and whether the verifier is learning evidence correctness rather than exploiting primary-label information?
2 comments
Follow-up: summary of clarifications and remaining questions
Thank you again for your constructive review and for recognizing the value of the conditional-reliability perspective. We would appreciate your feedback on our response above. For convenience, we summarize the clarifications corresponding to your four questions.
1. Scope of the CEVD evaluation. The evidence-drift analysis spans three datasets and two model families, whereas CEVD is evaluated as a proof of concept on Mistral–ESConv with one verifier. We distinguish these scopes rather than treating the broader diagnostic results as evidence that CEVD generalizes across all settings.
2. Selective baselines. The random-deferral comparison matches the retained count exactly. The reported entropy and disagreement comparisons have different realized coverages, so they do not establish superiority at exactly matched coverage. We also distinguish evidence replacement by Hybrid-Compose from using expert confidence to select the original evidence predictions.
3. Interpretation of CEDR. We agree that CEDR contains no independent information beyond primary accuracy and JEM. Its value is to make the conditional evaluation target explicit. The arithmetic example in our response illustrates how primary accuracy and JEM can both improve while conditional evidence error increases. We therefore advocate reporting CEDR alongside those metrics and the conditioning-set size, not replacing them.
4. Verifier conditioning. For a fixed verifier, threshold, and eligibility rule, replacing predicted primary labels with gold labels leaves scores and retention decisions unchanged on primary-correct instances, where those labels already coincide. This pointwise equality does not eliminate possible effects on development-set threshold selection, overall coverage, or learning during training, nor does it establish that the verifier avoids reliance on label priors.
Could you let us know whether these clarifications address your questions about the evaluation objective and verifier conditioning, and which concerns remain most important for your assessment? Please also let us know if there are additional points that would benefit from clarification.
Thank you for your time and consideration.
Response to 28YF: evaluation value, selective baselines, and verifier conditioning
Thank you for recognizing the evaluation's practical value and transparent scope. We address your four questions below, separating reported results from mathematical clarification.
1. Scope and magnitude of the CEVD result.
The diagnostic study covers three datasets and two model families; the CEVD intervention is evaluated only on Mistral-ESConv with one verifier. We do not infer cross-dataset or cross-backbone effectiveness from the diagnostic results.
Table 3 and Appendix H report two development-selected operating points. Rates are percentages; Capture is the fraction of original drift cases withheld.
The reductions are 3.80 and 6.04 percentage points at reduced evidence coverage. Primary predictions and retained evidence labels remain unchanged. These existing results demonstrate targeted selective release in one setting, not broad method generalization. CEVD on Qwen and on MELD/RECCON remains untested. We maintain the distinction between the multi-setting diagnostic contribution and this mitigation proof of concept.
2. Stronger baselines at exactly matched coverage.
The submitted comparison with random deferral matches the number of retained predictions exactly. Its empirical p=0.0002 concerns that control only. Appendix H also reports entropy and disagreement, but their realized coverages differ. For example, at the 90% development target, coverage/sCEDR are 89.33/32.57 for support, 88.78/33.20 for entropy, and 89.83/32.97 for disagreement. These results do not establish superiority at matched coverage.
The requested expert-confidence, margin, and sampled self-consistency comparisons are not reported. Hybrid-Compose replaces evidence labels and is not an expert-confidence deferral baseline; the reported disagreement score is not a substitute for sampled self-consistency. A fair comparison must keep the original predictions fixed, score their original evidence, retain the same number under the same eligibility rule, and report both PCov and sCEDR. Offline ranking must be distinguished from development-threshold transfer. Our superiority claim is limited to matched random deferral.
3. What CEDR adds to reporting JEM and primary accuracy.
We agree that CEDR adds no independent information when JEM and primary accuracy are both available. Its purpose is to state the conditional evaluation target explicitly, not to introduce a new probability identity.
Consider an arithmetic illustration, not an experimental result, with 1,000 instances per system:
Both primary accuracy and JEM improve in B, but evidence error conditional on primary correctness increases from 10% to 15%. B is better for full-pair success, while A has lower conditional error. This distinction is fully recoverable from the original two metrics; explicitly reporting CEDR makes the objective tradeoff visible. It is not a reason to rank systems by CEDR alone, because their conditioning subsets can differ. Section 3.2 therefore reports it with primary accuracy and conditioning-set size.
4. Gold-primary training versus predicted-primary inference.
For a fixed verifier, define S as the instances where the generated primary label is correct. For every eligible instance in S, the following equality holds:
pϕ(e^i∣Xi,m^i)=pϕ(e^i∣Xi,mi).
Thus, substituting gold for predicted primary labels has zero direct score effect on this subset; with a fixed threshold and eligibility gate, retention is identical. This is an algebraic statement, not a measured ablation. Test references identify S only for evaluation, not for release decisions.
It does not make the overall mismatch irrelevant. Primary errors can affect development risks and hence the selected threshold, overall coverage, and behavior outside S. Nor does it imply that training on predicted labels would learn the same verifier.
Equation (5) learns a conditional evidence-label distribution, not a binary correctness classifier. Primary-label information is intentionally used; what remains unisolated is the contribution of dialogue beyond label priors. Context-only and primary-only controls, alongside training with out-of-fold predicted primary labels, would distinguish these effects without using test labels. We do not equate selective error reduction with proof that the verifier avoids label-prior shortcuts.