- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces INTERVIEWVALUES-100K, a corpus of 100,000 single- and multi-turn English interview units drawn from MediaSum, NPR-2P, Tennis Interviews, and NewsInterview, annotated with machine-generated labels for 20 Schwartz-based value categories, together with retained source records and answer-side evidence spans. It also proposes SRLCF, a dual-encoder framework that combines MPNet semantic matching between dialogue and value definitions with a DeBERTa direct classifier, fused by label-specific gates, plus an answer-marker residual and a dialogue-level value representation trained with fine-value and higher-order auxiliary supervision. SRLCF reaches 0.510 Macro-F1 on a development benchmark versus 0.483 for DeBERTa+ASL, and 0.494 versus 0.475 on a 1,000-dialogue human-annotated MediaSum set, with a paired 95% CI spanning zero. Representation analyses show fine-value supervision improves frozen linear probing (+0.0074 Macro-F1) without improving fused classification, and that value-centroid distances correlate with Schwartz circular distances (ρ = 0.502).
Critical strengths
A source-linked, evidence-annotated corpus for a genuinely under-resourced task. INTERVIEWVALUES-100K retains source identifiers, dialogue turns, and answer-side evidence spans, and it distinguishes question context from answer evidence. This is a meaningful resource contribution even if the labels require later audit.
An honest separation of representation quality from fused classification. The finding that fine-value supervision improves a frozen linear probe (+0.0074 Macro-F1) but not fused classification (Δ = −0.0003, all CIs spanning zero) is a useful negative result that is cleanly reported and correctly interpreted.
Critical weaknesses
No clean confirmatory evaluation supports the comparative method claim. The development test split is explicitly labeled a development benchmark because seed-42 diagnostics informed architecture development. The human evaluation is retrospective, positive-only, single frozen run per system, and the paired Macro-F1 difference is +0.020 with 95% CI [−0.012, +0.052], which crosses zero. The paper itself states this “leaves comparative uncertainty.” There is therefore no evaluation in the paper that establishes that SRLCF outperforms DeBERTa+ASL.
The 100k training labels are unaudited and cannot be treated as ground truth. The MediaSum portion is annotated by a single LLM family (qwen3.8-max) with two prompt variants and a third strict pass; agreement is defined as same-model-channel agreement, which is not semantic validation. No expert human override was applied, and no human audit of the training corpus was performed. The human evaluation uses different dialogues and does not validate these labels. This limits the corpus’s value as a benchmark and makes it unclear how much of the development-benchmark gain reflects annotation style rather than value understanding.
The human annotation protocol has no reported inter-annotator agreement, and the hierarchy auxiliary contradicts itself. The A/B decisions were not retained, so IAA cannot be reconstructed; annotators are 20 master’s students with no value-theory or English-proficiency screening; and the positive-only population cannot measure false alarms or prevalence. Separately, the paper reports that removing the hierarchy auxiliary improves the base configuration by +0.0024, yet the loss is retained in the final model; this inconsistency is not resolved.
Questions for authors
Can the authors add a human audit of a stratified sample of the 100k training labels (by source, rare label, polarity, question-only cue), with reported agreement against the machine labels? This is essential for treating the corpus as a benchmark.
Can the authors report inter-annotator agreement on the 1,000 human-annotated dialogues, or reconstruct it from retained A/B records?
Can the authors run a genuinely confirmatory comparison—e.g., a new model trained after all architecture decisions are frozen, evaluated on the retrospective human set and on a new blind sample—and report the paired interval? Without this, the comparative claim remains unsupported.
Why is the hierarchy auxiliary retained when removing it improves the base configuration by +0.0024? Can the authors either explain the choice or drop the loss?
Given that fine-value supervision improves frozen probing but not fused classification, what is the intended takeaway of SRLCF as a method? Is the contribution the representation, the fused classifier, or the corpus?
The paper states that checkpoints were deleted after inference. Can the authors clarify which artifacts are actually released, and whether the human-eval comparator rerun and the deployment checkpoint can be used to reproduce the main tables?