- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper audits instruction and preference datasets using a scoring LLM's in-context behavior. A directed semantic nearest-neighbor graph measures how neighboring demonstrations change the likelihood of a target response or preferred-option token. Incoming and outgoing influence scores select anomalous records for LLM arbitration against contrasting neighbors, while a degree-based uniqueness guardrail protects poorly connected prompts from deletion. The Shapley connection is a singleton truncation of a local in-context game, empirically compared with multi-shot Shapley, rather than a guarantee about retraining-based data valuation.
HelpSteer2 and HH-RLHF experiments show positive rank correlation with estimated multi-shot Shapley, enrichment of arbitrator-confirmed inconsistencies, and distinctive DPO loss and gradient behavior. The test audit uses three arbitrator families. Retraining after filtering or repair yields approximately 0.5–0.6 percentage points at best on full evaluation splits, with larger improvements on small corrected inversion subsets. The evidence supports useful model-relative auditing signals more strongly than independently established label correctness or general alignment improvement.
Critical strengths
-
The paper identifies and tests a useful distinction between incoming contradiction and outgoing demonstration quality. Equations (1)–(5) and Section 3.3 turn this distinction into a concrete, inference-only audit procedure. In the controlled six-slice study, both incoming-score tails have higher anchor-fault rates than random sampling (32.1% and 36.6% versus 19.7%), whereas outgoing-score tails do not (Table 9, p. 21). The HelpSteer2 comparison also holds the arbitrator and Stage-2 matching procedure fixed, obtaining 18/70 versus 4/70 confirmed contradictions (Section 4.2). These controls support a practical contribution beyond simply using an LLM judge, and the reported 9.69 A100 GPU-hours for full HH-RLHF scoring gives the approach a plausible corpus-scale operating point.
-
Complementary analyses substantiate a meaningful phenomenon. The 200-target Monte Carlo comparison tests multi-shot valuation fidelity; exact LoRA gradients show a 56.7% versus 37.0% early conflict rate for confirmed versus clean records, with matched checks (Section 4.3; Table 12). Three-provider evaluation and scorer, retriever, neighborhood, and budget ablations strengthen the audit evidence. Downstream retraining includes a metric-only arm, and the limitations explicitly acknowledge single-arbitrator labeling and limited full-training replication.
Critical weaknesses
-
Arbitrator agreement is established more directly than objective label correctness. The training audit and repaired test labels primarily use Gemini judgments; the three-provider check obtains 60%–68% pairwise raw agreement and 48% three-way agreement (Section 4.4; Appendix B, pp. 21–22). Model–arbitrator agreement and failure to fit preferences can reflect shared model preferences or legitimate annotation differences. The acknowledged limitation does not itself invalidate the contribution, but statements that the benchmark penalizes the better answer and that confirmed records are definitively corrupted exceed independent validation. This matters when changing human labels, including safety-related ones. Blinded, rubric-based human adjudication of stratified flagged, cleared, and random samples would clarify correctness and disagreement; alternatively, consistently restrict claims to model-relative inconsistency. Supply the exact HHH rubric and preference prompt: Appendix A.1 contains placeholders despite Appendix A.5.2 stating that it includes HHH criteria.
-
The small full-retraining gain needs stronger control and uncertainty estimates. Table 2 reports 0.51–0.59-point gains with one from-scratch run per comparable arm; Sanitized uses another trainer build. Table 13 adds a Repaired run whose seed and build change together. The useful five-seed experiment in Table 14 resumes from one step-1,000 checkpoint, establishing a continuation benefit rather than full-training variance. Its full-split mean gain is 0.15 points; larger, separated gains concern 142 corrected inversions. Thus the evidence does not yet establish a robust 0.5–0.6-point full-run gain or isolate targeted pruning from reduced data volume. Use identical trainers, independent full-training seeds, a size-matched random-deletion control, and paired evaluation uncertainty, or narrow the quantitative claims. Also remove or substantiate Appendix B's assertion that unflipped majority errors “can only understate” gains: common labels alone do not establish the direction of their effect on differences between arms.
Questions for authors
-
Is any blinded human adjudication available for the flagged and control samples? Please give the exact HHH instructions and the actual HH-RLHF zero-/one-shot scoring prompt, including option formatting. Are option positions fixed consistently across each record's baseline, probes, and arbitration, and has option-order sensitivity been checked? Appendix A.1's placeholders and Appendix A.5.2's brief “Harmlessness > Helpfulness” reference do not fully specify the operative rubric.
-
What do paired uncertainty estimates show for the Table 2 full-split differences under the same trainer build? Are matched-build, independent from-scratch replications or a size-matched random-pruning control available? Please distinguish those results from the conditional step-1,000 continuation experiment when stating the main gain.
-
How are multi-shot demonstration order, context-length overflow, and the 20 sampled permutations handled in Section 4.1? How are repeated demonstrations aggregated across targets into the 1,960-record comparison, and what is the uncertainty on the reported correlations? Also specify how Equation (4) handles zero neighborhood variance and undefined outgoing scores. These details determine whether the reported rank agreement is reproducible and what the “first-order” approximation actually covers.
-
At what matched selection budget or threshold are the zero-shot likelihood and margin filters said to miss over 99% of Regime I records in Table 3? Please report their overall audit yield on the same adjudicated pool, alongside their Regime I recall. This would clarify the magnitude of improvement over these much cheaper filters without conditioning the headline comparison only on the proposed method's distinctive slice.
3 comments
[2/3] Response to Reviewer pw89
Q1: Blinded human adjudication, exact HHH prompt, and option-order consistency
In Appendix A.1, we now provide the verbatim HH-RLHF zero-shot and one-shot scoring prompt, including the exact HHH rubric instruction and option template. Each record's two responses are assigned to Option 1 and Option 2 once using a deterministic hash of the conversation context, ensuring that option positions remain strictly identical across that record's zero-shot baseline, one-shot probes, every appearance as an in-context demonstration, and Stage-2 arbitration. To verify option-order invariance, we re-arbitrated all 448 evaluation-split pairs after swapping Option 1 and Option 2 in both records (Appendix B), finding 90.0% verdict agreement () and 92.1% directional agreement on flagged pairs, matching same-order repeat runs (90.9%). For blinded human adjudication, 17 raters provided 314 HHH-rubric ratings on 100 HH-RLHF test pairs (50 Confirmed, 25 Cleared, and 25 Random) without seeing the recorded benchmark label (Appendix Table 13). Strict human majorities confirm defects on 66.0% of single-arbitrator confirmed pairs, rising to 76.0% ( majority) and 88.9% ( unanimous) versus 56.0% on controls, prefer our flipped label on 78.6% of Repaired flips with unequal directional votes versus 47.6% on controls (), and uphold the original HH-RLHF label on only 10.0% of Confirmed pairs versus 36.0%–40.0% on controls (). Together with our model-relative framing, this indicates that our flagged pairs (especially under multi-arbitrator consensus) are predominantly genuine defects or ambiguous ties rather than valid preferences.
Q2: Paired uncertainty estimates, matched-build from-scratch replications, and random-pruning controls
Under the identical trainer build and setup across three independent from-scratch seeds per arm (Table 2 and Appendix Table 15), while full-split accuracy differences are modest and within seed-level variance, every curated arm outperforms Uncleaned on both inversion splits (with all 12/12 runs beating their matched uncleaned seed on majority inversions), whereas size-matched random deletion lowers mean inversion accuracy:
- Full splits (Repaired / Sanitized vs. Uncleaned): 68.77 ± 0.26% / 68.88 ± 0.17% vs. 68.66 ± 0.19% on Full Raw (+0.11% / +0.22%), and 69.59 ± 0.28% / 69.68 ± 0.19% vs. 69.38 ± 0.18% on Repaired Eval (+0.21% / +0.30%, seed-level ). Item-level paired bootstrap 95% CIs likewise include zero and McNemar tests give (Appendix B).
- Test inversions ( / Majority ): +2.11%–6.10% over Uncleaned on single-arbitrator inversions (Metric-Comb accuracy , Repaired margin ) and +2.41%–4.81% on majority inversions (Metric-only accuracy , Repaired margin ), vs. −0.34% to −1.89% across both size-matched random-deletion controls.
In Section 4.5, we state our main results strictly from these end-to-end (steps ) runs, keeping the shared-checkpoint continuation runs (steps , Appendix Table 16) separate.
Q3: Multi-shot Shapley details in Section 4.1 and Equation (4) edge cases
We added these reproducibility details to Sections 3.2 and 4.1 and Appendix A.1:
-
Multi-shot permutation protocol and overflow handling (Section 4.1 and Appendix A.1): For each of the 200 targets, we sample 20 random permutations of its 15 incoming neighbors, append demonstrations in permutation order, and average each neighbor's marginal log-likelihood gain across the 20 permutations. On HelpSteer2, all full 15-neighbor prompts fit within a 24,000-token limit without truncation. On HH-RLHF (Appendix A.1), if a coalition exceeds the 8,192-token context window, the scorer keeps the BOS token and left-truncates older prefix tokens so the newly added demonstration and target query at the end stay intact.
-
Aggregation into unique demonstrations and bootstrap CIs (Section 4.1): We report bootstrap 95% confidence intervals on the per-target Spearman rank correlations across all 200 targets (median [95% CI 0.63, 0.73], mean [0.63, 0.69]), and clarify that the 1,960 demonstrations in Table 1 correspond to the unique neighbor nodes across the 3,000 directed edges, with repeated appearances averaged per demonstration node.
-
Equation (4) edge cases (Section 3.2): We specify that neighborhood standardization sets the advantage to zero whenever neighborhood variance is zero and omits any neighbor with zero outgoing edges () from reference statistics.
[3/3] Response to Reviewer pw89
Q4: Matched selection budget and overall audit yield for zero-shot likelihood and margin filters
We agree that reporting only Regime I recall in the original Table 3 conditioned the comparison on our method's distinctive slice rather than overall audit yield. In that table, we used a matched budget of 8,500 records (top 5.3% of the corpus, matching our pool) for both zero-shot filters: the top 8,500 records ranked by Zero-Shot Likelihood and Zero-Shot Margin include only 3 and 7 of the 968 Regime I errors confirmed in our left tail, missing 99.7% and 99.3%.
To evaluate overall audit yield without conditioning on our pool, we also ran full Stage-2 arbitration directly on all 8,500 candidates selected by Zero-Shot Likelihood and Zero-Shot Margin (Section 4.3 and Appendix A.2, Table 4). Because our original Stage-1 score (41.4% overall confirmed) depends only on the mean neighbor shift (Eq. 2), it flags pairs whose label flips under neighbor context, but misses obvious zero-shot outliers whose low likelihood stays unchanged under neighbors (). To capture both zero-shot outliers and neighborhood contradictions, we updated the Stage-1 rule in Section 3.3 to combine and via (Metric-Comb at , where every improves over across in Appendix Table 4):
-
With the unaligned base scorer (Qwen3.5-9B-Base): Because the base model's zero-shot prior is not yet calibrated to HHH preferences (50.1% and 50.9% confirmed under paired twin arbitration), conditioning on the 15 nearest neighbors draws a local preference signal from the corpus. Adding our in-context log-likelihood shift to via Metric-Comb lifts confirmed contradictions to 53.8%, outperforming both Zero-Shot Likelihood and Zero-Shot Margin (Fisher over Zero-Shot Margin). Thus, in-context conditioning already improves over zero-shot filtering even on an unaligned base scorer.
-
With the post-trained scorer (Qwen3.5-9B): When general HHH preferences are already encoded in , Zero-Shot Likelihood and Zero-Shot Margin reach 56.5% and 59.1% confirmed under the same paired twin arbitration. Combining with via Metric-Comb reaches 62.4% confirmed while capturing three times as many negative-shift contradictions ( within each method's 8,500-pair budget: 1,452 vs. 390 and 494). Thus, even when a post-trained scorer makes zero-shot filtering strong on obvious single-record outliers, in-context conditioning adds complementary recall on local convention clashes that look plausible in isolation.
[1/3] Response to Reviewer pw89
We thank Reviewer pw89 for the thorough, constructive review and positive assessment of our distinction between incoming and outgoing influence, controlled slice studies, and LoRA gradient conflict analysis. Below we address each of your points and questions, with all corresponding updates highlighted in blue in the revised PDF.
W1: Arbitrator agreement versus objective label correctness, model-relative framing, and exact prompts
Thank you for this feedback. We have addressed each part of this point in the revised manuscript:
Cross-arbitrator agreement and consensus evaluation (Section 4.4, Table 2, and Appendix B): We report chance-corrected binary agreement (pairwise Cohen's κ=0.14-0.28, Fleiss' κ=0.21) alongside directional agreement when two providers both flag a pair (86.2%–98.3%) across the 448 evaluation-split pairs. Because single-model verdicts can reflect model-specific preferences on borderline cases, Table 2 also evaluates the ≥2/3 majority-confirmed inversions (n=97) separately.
Model- and arbitrator-relative claims (Sections 4.3, 4.4, and Appendices A.6–A.7): We revised the specific statements noted, including in Section 4.3 ("arbitrator-confirmed records capture systematic preference inconsistencies"), Section 4.4 ("penalizes responses preferred by both the LLM arbitrators and the evaluated models"), and the Appendix qualitative examples. To test against independent human judgments directly, we also conducted a blinded, rubric-based human adjudication on 100 HH-RLHF test pairs (50 Confirmed, 25 Cleared, and 25 Random) with 314 ratings from 17 raters (see Q1 and Appendix Table 13). Consistent with our model-relative framing, human raters show moderate four-way and binary-defect agreement (α=0.31-0.36) and flag substantial baseline noise even on random HH-RLHF pairs (56.0%), while upholding the original benchmark label on only 10.0% of single-arbitrator confirmed pairs and 0% of unanimous pairs versus 36.0%–40.0% on controls.
Exact HH-RLHF rubric and scoring prompt (Appendix A.1): We replaced the generic placeholder box in Appendix A.1 with the verbatim HHH rubric instruction and option-formatting template used in our code (see Q1 for option-position fixing and option-swap verification).
W2: Full-retraining controls, matched trainer builds, and Appendix B unflipped-errors claim
We appreciate these methodological suggestions and incorporated all of them into Section 4.5 and Appendix B:
Identical trainer build, independent from-scratch seeds, and size-matched random deletion: We ran three independent from-scratch DPO seeds per primary arm under the identical trainer build and setup, together with size-matched random-deletion controls (Table 2 and Appendix Table 15).
Scoped downstream claims, paired uncertainty, and separation from step-1,000 continuation: In the Abstract, Introduction, Section 4.5, and Section 5, we clearly separate the from-scratch runs (Table 2) from the five-seed step-1,000 continuation study (Appendix Table 16), report paired seed-level and item-level uncertainty (see Q2), and scope our downstream claims accordingly. Because our arbitrated arms modify only 2.4% of the training corpus (5.2% for the metric-only arms), full-split accuracy differences across independent seeds are modest (~0.2–0.3 pp, within seed-to-seed variance). Gains are clearer on the cleaned evaluation splits (Sanitized and Repaired) than on Full Raw, and are largest and statistically significant on the corrected test inversions and majority-confirmed inversions, where every curated arm improves over Uncleaned (including all 12/12 runs against matched uncleaned seeds on majority inversions) while random deletion lowers mean accuracy.
Appendix B unflipped-errors sentence: We removed the sentence in Appendix B stating that the unflipped majority test inversions "can only understate" the cleaned arms' gains.