- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper focuses on a meaningful problem in ECPE that the model tends to select ones that are close to the emotion clauses, which is also potentially beneficial to tasks with sequential input and sequential labeling. The proposed method aims to construct counterfactuals that are challenging to the predictor that that the model can better learn meaningful emotion-cause relationships rather than easily fall into clause proximity.
The authors use GPT-4o to generate three alternative cause clauses for each gold cause. A verifier is used to filter out low quality couterfactuals. Highest residual emotion-support score lead to the hardest-valid selection, which will be used for the subsequent DPO training.
Critical strengths
- The paper identifies a concrete weakness that models can leverage positional proximity as a shortcut, and may overlook genuinely meaningful emotion-support clauses.
- The paper reported comprehensive evaluations from diversive perspectives.
- The paper includes long-distance and position-rebalanced results, distractor relocation, independent hard negatives, ablations, threshold sensitivity, and cross-backbone results. The 7B, 14B, and Llama results are a useful check that the gains are not confined to one model size.
Critical weaknesses
-
My major concern is, the proposed hardest-valid selection does not establish that the selected negative is genuinely difficult for the target model. The paper defines counterfactual eligibility using a frozen verifier, with threshold and margin as 0.8 and 0.3. In other words, the hardest negative is defined as the candidate that retains the greatest emotion-support score according to the verifier. My problem is that this verifier uncertainty or residual support is not necessarily equivalent to the difficulty for the trained cause extractor. Therefore, the proposed hardest-valid selection effectively chooses the candidate with the smallest verifier-measured support reduction among candidates that pass the threshold. The central claim is that the method learns from especially challenging counterfactuals. Without validating the hardness measure against the target task, the improvement could instead arise from another property of the selected examples.
-
Another issue is that the human audit. The authors argue that their counterfactuals are challenging because they are plausible alternatives that compete with the annotated cause. However, according to Appendix D.4, the primary evaluation criteria is contextually plausibility rather than causally plausibility. The audit further shows that 94% of retained counterfactuals were not judged plausible as causes, even though 91.1% were contextually plausible. This raises concerns about whether these are genuinely challenging causal alternatives or merely fluent distractors. The authors might reasonably hypothesize that a negative can be difficult for a model even when humans recognize it as incorrect. That is true. The human audit alone does not prove the negatives are easy for models.
-
The verifier performs two essential functions: determining whether a candidate is eligible and selecting the hardest candidate. Thus, the quality of the training supervision depends directly on its ability to assess emotion support. Appendix B.3 describes a verifier that calculates a binary probability from the logits corresponding to the Chinese tokens. The paper specifies its quantization and inference configuration but does not provide a clearly identifiable public checkpoint or a direct evaluation establishing its competence at the intended verification task. More importantly, the reported manual audit does not substitute for a systematic assessment of the verifier's discriminative capability.
-
The experimental design does not fully disentangle the proposed mechanism from the benefits of additional synthetic supervision. Although the matched comparisons control the backbone, initialization, reference policy, and optimization configuration, CPA-DPO additionally benefits from GPT-4o-generated counterfactuals, verifier-based filtering, and augmented training documents. Consequently, the reported improvements may partly arise from additional semantic supervision or favorable training-data selection rather than the proposed preference-construction mechanism. A stronger comparison would include models trained with the same synthetic counterfactuals and comparable training exposure, without the proposed preference-selection procedure.
Questions for authors
See Weaknesses.
3 comments
Response to KE55 (2/2): Verifier Competence and Matched Synthetic-Data Controls
Response to KE55 (2/2): Verifier Competence and Matched Synthetic-Data Controls
- Public verifier identity and direct competence test.
The verifier is Qwen/Qwen2.5-32B-Instruct, frozen with 0 ECPE-specific training samples, epochs, or updates. One verifier is shared across folds; candidate generation/scoring uses only the corresponding training fold. It is separate from the trainable cause selector. Support is the binary softmax over final-position logits for single-token "是"/"否", with left padding, maximum length 2,048, NF4 double quantization, and BF16 compute. These details are explicit in the revised appendix.
We directly test its intended task: whether the specified emotion has causal support in the current document. From the pre-gate constructible pool, we sample 400 source pairs (40/fold; 100 same-clause, 300 different-clause), uniformly selecting one structurally admissible replacement per source without consulting scores. Original/replacement documents yield 800 views.
Two annotators independently judge Support/Non-support/Ambiguous, blinded to scores and retention status; a third adjudicates. The source quotation identifies the target; causal facts absent from the current document cannot count as support. We use the original verifier scoring procedure and a pre-fixed diagnostic threshold, without tuning on these labels.
The replacement views contain 232 Support, 148 Non-support, and 20 Ambiguous cases. On the 380 clear replacements:
| Human label | Predicted Support | Predicted Non-support |
|---|---|---|
| Support | 207 | 25 |
| Non-support | 18 | 130 |
Accuracy is 88.68%, recall 89.22%, specificity 87.84%, and balanced accuracy 88.53%. Same-/different-clause replacement balanced accuracies are 87.02%/89.07%. Across all views, 770 are clear and 30 ambiguous; balanced accuracy is 90.62%. We emphasize replacement-only results because originals are predominantly positive.
We separately audit whether the actual gate identifies replacements that substantially weaken support while preserving target/referent consistency:
| Human paired-intervention judgment | Gate pass | Gate fail |
|---|---|---|
| Clearly satisfies | 125 | 25 |
| Clearly fails | 9 | 195 |
| Ambiguous | 5 | 41 |
Among all 139 gate passes, 125 satisfy this criterion, 9 fail, and 5 are ambiguous. On clear judgments, precision is 93.28% and recall 83.33%. This evaluates support-changing interventions; it does not identify exact gold-cause membership or the final preference error rate.
- Separating preference construction from additional synthetic supervision.
We agree that Vanilla versus CPA alone does not isolate the contribution of synthetic augmentation. We therefore add controls with the same synthetic inputs or candidate resources:
- C1: CPA's exact augmented inputs and complete gold chosen outputs, trained with chosen-only cross-entropy.
- C2: the same inputs/chosen outputs, with DPO against a mapped generic original-document non-gold/non-emotion negative. CPA/C2 both use singleton rejected outputs.
- Random-valid: the same teacher pool, eligibility, sources, role/order variables, and dummy, selecting uniformly within the eligible pool.
- No candidate Drop gate: the same CPA source cohort and source threshold, removing only the candidate Drop criterion and maximizing residual support over structurally admissible candidates.
- C3: the same retained records and selected candidates, with gold-original/trap-appended placement fixed.
All use the same 6,178 source identities, 100% source-set overlap, fold-specific SFT starting policy, and 263 additional updates across ten folds. C1 has 6,178 chosen-only examples; DPO controls have 6,178 pairs. These match sources/examples/updates, without establishing equal token exposure or FLOPs.
| Method | Overall F1 | LONG F1 | HN accuracy (%) |
|---|---|---|---|
| Vanilla-DPO | 83.62 | 70.42 | 87.92 |
| C1: chosen-only augmented SFT | 84.12 | 72.25 | 88.98 |
| C2: same-input generic-negative DPO | 84.42 | 72.96 | 89.84 |
| Random-valid | 84.48 | 73.41 | 90.24 |
| No candidate Drop gate | 84.09 | 72.02 | 89.33 |
| C3: fixed placement | 85.06 | 74.50 | 90.87 |
| CPA-DPO | 85.43 | 75.46 | 91.35 |
C2 exceeds Vanilla by +0.80 Overall/+2.54 LONG, consistent with a benefit from augmented inputs. CPA adds +1.31/+3.21 over C1, +1.01/+2.50 over C2, +0.95/+2.05 over uniform eligible selection, and +1.34/+3.44 over removing the candidate Drop gate. These comparisons support gains beyond synthetic-input exposure under the tested controls; they are not an additive decomposition of component effects. C3 changes position and distance jointly.
Looking Forward to Further Discussion
Dear Reviewer KE55,
Thank you again for your careful review and constructive suggestions. We sincerely appreciate your recognition of the motivation and experimental scope of our work, as well as the opportunity to strengthen the empirical validation of our proposed method. We have carefully considered your concerns and provided detailed responses with additional experimental evidence in our rebuttal.
Regarding your concern about hardest-valid selection and target-model difficulty, we have added direct target-policy hardness diagnostics using the frozen pre-DPO ECE-SFT model. The results show that hardest-valid selection produces more competitive rejected responses than uniform random selection within the eligible pool. We have also provided candidate-level ranking analyses to further examine the relationship between verifier-based selection and target-policy difficulty.
Regarding counterfactual plausibility and preference validity, we have clarified the distinction between contextual plausibility, causal plausibility, and model difficulty. We have additionally conducted a 300-instance human audit of the final augmented inputs and provided independent hard-negative evaluations to assess the learned cause discrimination beyond the generated training counterfactuals.
Regarding verifier reliability, we have specified the exact public verifier checkpoint and its inference configuration, and conducted a direct human-labeled competence evaluation. These results provide additional evidence supporting the verifier's ability to assess emotion support and the reliability of the eligibility screening procedure.
Regarding the contribution beyond additional synthetic supervision, we have added several controlled experiments, including chosen-only augmented SFT (C1), same-input generic-negative DPO (C2), random-valid selection, and fixed-placement DPO (C3). These comparisons use matched source records and training-update budgets, providing further evidence for the benefits of our targeted preference construction beyond synthetic augmentation alone.
We hope these clarifications and additional experimental results address your main concerns. We would greatly appreciate your feedback on whether the new evidence has adequately addressed these issues, and please let us know if any concerns remain. We would be happy to provide further explanations and continue the discussion.
Thank you again for your time and consideration.
Response to KE55 (1/2): Target-Policy Hardness and Counterfactual Plausibility
Response to KE55 (1/2): Target-Policy Hardness and Counterfactual Plausibility
Dear Reviewer KE55,
Thank you for identifying the key distinction between verifier support and target-model difficulty. We have added direct target-policy diagnostics, a verifier competence test, and controls using the same synthetic resources to the revision. We address your first two concerns here and the remaining two in the next comment.
We agree that the two quantities need not coincide. Hardest-valid has the operational definition
j^*=\arg\max_{j\in V}A_{\mathrm{cf}}^{(j)}.$$ Ties follow the original candidate order. This is verifier-relative selection within the eligible pool. The revised text makes this scope explicit. We now evaluate difficulty using each fold's frozen ECE-SFT cause selector before DPO. On the actual augmented inputs, we score the complete structured chosen/rejected responses: $$m_j=L_{\mathrm{SFT}}(y^+\mid x_j^+)-L_{\mathrm{SFT}}(y_j^-\mid x_j^+).$$ $L$ sums completion-token log-probabilities, excluding the prompt and including EOS; lower $m_j$ means stronger competition from the rejected response. Random-valid and hardest-valid are matched by source, role orientation, within-block order, dummy, and input protocol, with candidate text allowed to differ. | Selection | Mean margin | Rejected response outscores chosen | |:--|--:|--:| | Uniform random eligible | 0.43 | 1,161/6,178 (18.79%) | | Highest eligible residual support | 0.31 | 1,372/6,178 (22.21%) | The direction also holds for the 5,994 singleton-gold records: margin 0.435 -> 0.315 and rejected-outscoring rate 18.70% -> 22.14%. Thus the result is not confined to multi-gold cases. These are pre-DPO completion-competition measurements; they are not free-generation error rates. Total log-probabilities can retain length effects. We additionally compare candidate rankings within the same source. The 4,328 sources with at least two eligible candidates contain 10,414 candidates and 7,844 candidate pairs; the remaining 1,850 sources offer no selection choice. Verifier residual-support rankings agree with target-policy difficulty rankings on 5,144/7,844 pairs (65.58%). The verifier selects the policy-hardest candidate in 2,501/4,328 sources (57.79%), compared with a 43.23% theoretical expectation for uniform selection, accounting for the mixture of two- and three-candidate pools. These results support residual support as an imperfect proxy that selects more competitive negatives on average. They do not establish that every selected candidate is the target policy's hardest negative. The downstream Random-valid comparison in the next comment separately tests whether this selection improves task performance. 2. Contextual plausibility versus causal plausibility. We agree that the original audit alone did not demonstrate model difficulty. Its 91.1% contextual plausibility and 6% causally plausible traps measure different properties: a coherent, event-related clause can remain an incorrect cause. Contextual naturalness supports the quality of the intervention; difficulty requires target-policy evidence such as the measurements above. We also audited final augmented preference inputs, separately from the original 180 replacement documents: 286/300 clearly valid, 14 ambiguous, and 0 clearly invalid. This evaluates sampled preference validity, not model difficulty. The human-audit and policy-diagnostic cohorts differ, so their aggregate rates do not establish how often a human-rejected trap also confuses the model. Independent hard-negative evaluation provides a further task-level check. It uses 2,000 held-out instances and 10,000 gold-negative comparisons. Negatives are original held-out clauses assessed for non-causality independently of the training teacher, verifier, and CPA candidate pool. The evaluated policy's yes/no log-odds rank the gold above the negative in 87.92% of comparisons for Vanilla-DPO and 91.35% for CPA-DPO. This supports improved cause discrimination beyond the generated training traps.