acceptodds
Under review as a conference paper at ICLR 2027

First-Order In-Context Shapley: Scalable Data Auditing for LLM Alignment and Evaluation

Abstract

Post-training alignment of Large Language Models (LLMs) is increasingly bottlenecked by data quality: massive preference and instruction-tuning corpora inevitably accumulate label noise, safety inversions, and inter-annotator contradictions that surface-level deduplication cannot detect and that are too costly to find by LLM-judging every record. While Data Shapley provides a principled framework for valuing individual records, exact computation requires intractable retraining, and fast embedding-space surrogates such as KNN-Shapley fail to separate high- from low-utility demonstrations ( with generative utility). We introduce an inference-only valuation pipeline that computes the first-order term of an in-context cooperative game over semantic -NN neighborhoods in forward passes without parameter updates, achieving positive rank correlation with multi-shot Monte-Carlo in-context Shapley on of targets (mean , close to the estimator's own split-half agreement of ) with less compute. By projecting pairwise log-likelihood shifts onto a directed influence graph, we characterize each record through three signals: outgoing Exemplifying Advantage, incoming Exemplified Advantage, and a topological One-of-a-Kind Score that protects rare tail records. Pre-filtering with these signals restricts arbitration to of records before routing each candidate and a contrasting twin to an LLM arbitrator, achieving enrichment in arbitrator-confirmed annotation errors on graded response-generation data (HelpSteer2) and on pairwise preference data (Anthropic HH-RLHF), including contradictions that zero-shot likelihood and margin filters miss. During DPO training, confirmed preference contradictions oppose the realized update far more often than clean records from early on ( vs. in exact LoRA gradients) and stay below chance in reward accuracy. Finally, auditing the HH-RLHF evaluation benchmark exposes label errors on which aligned models disagree with the benchmark of the time ( when three frontier arbitrator families unanimously agree), and retraining DPO from scratch after repairing or filtering the flagged training records improves reward accuracy by overall and on the repaired test inversions, even without the LLM arbitration step.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Dataset Auditing, Data Valuation, Shapley Value, Benchmark Integrity, LLM alignment, Direct Preference Optimization (DPO), RLHF, in-context learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pw89OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper audits instruction and preference datasets using a scoring LLM's in-context behavior. A directed semantic nearest-neighbor graph measures how neighboring demonstrations change the likelihood of a target response or preferred-option token. Incoming and outgoing influence scores select anomalous records for LLM arbitration against contrasting neighbors, while a degree-based uniqueness guardrail protects poorly connected prompts from deletion. The Shapley connection is a singleton truncation of a local in-context game, empirically compared with multi-shot Shapley, rather than a guarantee about retraining-based data valuation.

HelpSteer2 and HH-RLHF experiments show positive rank correlation with estimated multi-shot Shapley, enrichment of arbitrator-confirmed inconsistencies, and distinctive DPO loss and gradient behavior. The test audit uses three arbitrator families. Retraining after filtering or repair yields approximately 0.5–0.6 percentage points at best on full evaluation splits, with larger improvements on small corrected inversion subsets. The evidence supports useful model-relative auditing signals more strongly than independently established label correctness or general alignment improvement.

Critical strengths
  1. The paper identifies and tests a useful distinction between incoming contradiction and outgoing demonstration quality. Equations (1)–(5) and Section 3.3 turn this distinction into a concrete, inference-only audit procedure. In the controlled six-slice study, both incoming-score tails have higher anchor-fault rates than random sampling (32.1% and 36.6% versus 19.7%), whereas outgoing-score tails do not (Table 9, p. 21). The HelpSteer2 comparison also holds the arbitrator and Stage-2 matching procedure fixed, obtaining 18/70 versus 4/70 confirmed contradictions (Section 4.2). These controls support a practical contribution beyond simply using an LLM judge, and the reported 9.69 A100 GPU-hours for full HH-RLHF scoring gives the approach a plausible corpus-scale operating point.

  2. Complementary analyses substantiate a meaningful phenomenon. The 200-target Monte Carlo comparison tests multi-shot valuation fidelity; exact LoRA gradients show a 56.7% versus 37.0% early conflict rate for confirmed versus clean records, with matched checks (Section 4.3; Table 12). Three-provider evaluation and scorer, retriever, neighborhood, and budget ablations strengthen the audit evidence. Downstream retraining includes a metric-only arm, and the limitations explicitly acknowledge single-arbitrator labeling and limited full-training replication.

Critical weaknesses
  1. Arbitrator agreement is established more directly than objective label correctness. The training audit and repaired test labels primarily use Gemini judgments; the three-provider check obtains 60%–68% pairwise raw agreement and 48% three-way agreement (Section 4.4; Appendix B, pp. 21–22). Model–arbitrator agreement and failure to fit preferences can reflect shared model preferences or legitimate annotation differences. The acknowledged limitation does not itself invalidate the contribution, but statements that the benchmark penalizes the better answer and that confirmed records are definitively corrupted exceed independent validation. This matters when changing human labels, including safety-related ones. Blinded, rubric-based human adjudication of stratified flagged, cleared, and random samples would clarify correctness and disagreement; alternatively, consistently restrict claims to model-relative inconsistency. Supply the exact HHH rubric and preference prompt: Appendix A.1 contains placeholders despite Appendix A.5.2 stating that it includes HHH criteria.

  2. The small full-retraining gain needs stronger control and uncertainty estimates. Table 2 reports 0.51–0.59-point gains with one from-scratch run per comparable arm; Sanitized uses another trainer build. Table 13 adds a Repaired run whose seed and build change together. The useful five-seed experiment in Table 14 resumes from one step-1,000 checkpoint, establishing a continuation benefit rather than full-training variance. Its full-split mean gain is 0.15 points; larger, separated gains concern 142 corrected inversions. Thus the evidence does not yet establish a robust 0.5–0.6-point full-run gain or isolate targeted pruning from reduced data volume. Use identical trainers, independent full-training seeds, a size-matched random-deletion control, and paired evaluation uncertainty, or narrow the quantitative claims. Also remove or substantiate Appendix B's assertion that unflipped majority errors “can only understate” gains: common labels alone do not establish the direction of their effect on differences between arms.

Questions for authors
  1. Is any blinded human adjudication available for the flagged and control samples? Please give the exact HHH instructions and the actual HH-RLHF zero-/one-shot scoring prompt, including option formatting. Are option positions fixed consistently across each record's baseline, probes, and arbitration, and has option-order sensitivity been checked? Appendix A.1's placeholders and Appendix A.5.2's brief “Harmlessness > Helpfulness” reference do not fully specify the operative rubric.

  2. What do paired uncertainty estimates show for the Table 2 full-split differences under the same trainer build? Are matched-build, independent from-scratch replications or a size-matched random-pruning control available? Please distinguish those results from the conditional step-1,000 continuation experiment when stating the main gain.

  3. How are multi-shot demonstration order, context-length overflow, and the 20 sampled permutations handled in Section 4.1? How are repeated demonstrations aggregated across targets into the 1,960-record comparison, and what is the uncertainty on the reported correlations? Also specify how Equation (4) handles zero neighborhood variance and undefined outgoing scores. These details determine whether the reported rank agreement is reproducible and what the “first-order” approximation actually covers.

  4. At what matched selection budget or threshold are the zero-shot likelihood and margin filters said to miss over 99% of Regime I records in Table 3? Please report their overall audit yield on the same adjudicated pool, alongside their Regime I recall. This would clarify the magnitude of improvement over these much cheaper filters without conditioning the headline comparison only on the proposed method's distinctive slice.

3 comments
AuthorsOpenReview ↗

[1/3] Response to Reviewer pw89

We thank Reviewer pw89 for the thorough, constructive review and positive assessment of our distinction between incoming and outgoing influence, controlled slice studies, and LoRA gradient conflict analysis. Below we address each of your points and questions, with all corresponding updates highlighted in blue in the revised PDF.

W1: Arbitrator agreement versus objective label correctness, model-relative framing, and exact prompts

Thank you for this feedback. We have addressed each part of this point in the revised manuscript:

  • Cross-arbitrator agreement and consensus evaluation (Section 4.4, Table 2, and Appendix B): We report chance-corrected binary agreement (pairwise Cohen's κ=0.14-0.28\kappa = 0.14\text{-}0.28, Fleiss' κ=0.21\kappa = 0.21) alongside directional agreement when two providers both flag a pair (86.2%–98.3%) across the 448 evaluation-split pairs. Because single-model verdicts can reflect model-specific preferences on borderline cases, Table 2 also evaluates the ≥2/3\ge 2/3 majority-confirmed inversions (n=97n = 97) separately.

  • Model- and arbitrator-relative claims (Sections 4.3, 4.4, and Appendices A.6–A.7): We revised the specific statements noted, including in Section 4.3 ("arbitrator-confirmed records capture systematic preference inconsistencies"), Section 4.4 ("penalizes responses preferred by both the LLM arbitrators and the evaluated models"), and the Appendix qualitative examples. To test against independent human judgments directly, we also conducted a blinded, rubric-based human adjudication on 100 HH-RLHF test pairs (50 Confirmed, 25 Cleared, and 25 Random) with 314 ratings from 17 raters (see Q1 and Appendix Table 13). Consistent with our model-relative framing, human raters show moderate four-way and binary-defect agreement (α=0.31-0.36\alpha = 0.31\text{-}0.36) and flag substantial baseline noise even on random HH-RLHF pairs (56.0%), while upholding the original benchmark label on only 10.0% of single-arbitrator confirmed pairs and 0% of unanimous pairs versus 36.0%–40.0% on controls.

  • Exact HH-RLHF rubric and scoring prompt (Appendix A.1): We replaced the generic placeholder box in Appendix A.1 with the verbatim HHH rubric instruction and option-formatting template used in our code (see Q1 for option-position fixing and option-swap verification).

W2: Full-retraining controls, matched trainer builds, and Appendix B unflipped-errors claim

We appreciate these methodological suggestions and incorporated all of them into Section 4.5 and Appendix B:

  • Identical trainer build, independent from-scratch seeds, and size-matched random deletion: We ran three independent from-scratch DPO seeds per primary arm under the identical trainer build and setup, together with size-matched random-deletion controls (Table 2 and Appendix Table 15).

  • Scoped downstream claims, paired uncertainty, and separation from step-1,000 continuation: In the Abstract, Introduction, Section 4.5, and Section 5, we clearly separate the from-scratch runs (Table 2) from the five-seed step-1,000 continuation study (Appendix Table 16), report paired seed-level and item-level uncertainty (see Q2), and scope our downstream claims accordingly. Because our arbitrated arms modify only 2.4% of the training corpus (5.2% for the metric-only arms), full-split accuracy differences across independent seeds are modest (~0.2–0.3 pp, within seed-to-seed variance). Gains are clearer on the cleaned evaluation splits (Sanitized and Repaired) than on Full Raw, and are largest and statistically significant on the corrected test inversions and majority-confirmed inversions, where every curated arm improves over Uncleaned (including all 12/12 runs against matched uncleaned seeds on majority inversions) while random deletion lowers mean accuracy.

  • Appendix B unflipped-errors sentence: We removed the sentence in Appendix B stating that the unflipped majority test inversions "can only understate" the cleaned arms' gains.

AuthorsOpenReview ↗

[2/3] Response to Reviewer pw89

Q1: Blinded human adjudication, exact HHH prompt, and option-order consistency

In Appendix A.1, we now provide the verbatim HH-RLHF zero-shot and one-shot scoring prompt, including the exact HHH rubric instruction and option template. Each record's two responses are assigned to Option 1 and Option 2 once using a deterministic hash of the conversation context, ensuring that option positions remain strictly identical across that record's zero-shot baseline, one-shot probes, every appearance as an in-context demonstration, and Stage-2 arbitration. To verify option-order invariance, we re-arbitrated all 448 evaluation-split pairs after swapping Option 1 and Option 2 in both records (Appendix B), finding 90.0% verdict agreement (κ=0.79\kappa = 0.79) and 92.1% directional agreement on flagged pairs, matching same-order repeat runs (90.9%). For blinded human adjudication, 17 raters provided 314 HHH-rubric ratings on 100 HH-RLHF test pairs (50 Confirmed, 25 Cleared, and 25 Random) without seeing the recorded benchmark label (Appendix Table 13). Strict human majorities confirm defects on 66.0% of single-arbitrator confirmed pairs, rising to 76.0% (≥2/3\ge 2/3 majority) and 88.9% (3/33/3 unanimous) versus 56.0% on controls, prefer our flipped label on 78.6% of Repaired flips with unequal directional votes versus 47.6% on controls (p=0.035p = 0.035), and uphold the original HH-RLHF label on only 10.0% of Confirmed pairs versus 36.0%–40.0% on controls (p≤0.011p \le 0.011). Together with our model-relative framing, this indicates that our flagged pairs (especially under multi-arbitrator consensus) are predominantly genuine defects or ambiguous ties rather than valid preferences.

Q2: Paired uncertainty estimates, matched-build from-scratch replications, and random-pruning controls

Under the identical trainer build and setup across three independent from-scratch seeds per arm (Table 2 and Appendix Table 15), while full-split accuracy differences are modest and within seed-level variance, every curated arm outperforms Uncleaned on both inversion splits (with all 12/12 runs beating their matched uncleaned seed on majority inversions), whereas size-matched random deletion lowers mean inversion accuracy:

  • Full splits (Repaired / Sanitized vs. Uncleaned): 68.77 ± 0.26% / 68.88 ± 0.17% vs. 68.66 ± 0.19% on Full Raw (+0.11% / +0.22%), and 69.59 ± 0.28% / 69.68 ± 0.19% vs. 69.38 ± 0.18% on Repaired Eval (+0.21% / +0.30%, seed-level p=0.16–0.41p = 0.16\text{--}0.41). Item-level paired bootstrap 95% CIs likewise include zero and McNemar tests give p≥0.20p \ge 0.20 (Appendix B).
  • Test inversions (n=142n = 142 / Majority n=97n = 97): +2.11%–6.10% over Uncleaned on single-arbitrator inversions (Metric-Comb accuracy p=0.027p = 0.027, Repaired margin p=0.006p = 0.006) and +2.41%–4.81% on majority inversions (Metric-only accuracy p=0.010p = 0.010, Repaired margin p=0.028p = 0.028), vs. −0.34% to −1.89% across both size-matched random-deletion controls.

In Section 4.5, we state our main results strictly from these end-to-end (steps 0→1,2000 \to 1{,}200) runs, keeping the shared-checkpoint continuation runs (steps 1,000→1,2541{,}000 \to 1{,}254, Appendix Table 16) separate.

Q3: Multi-shot Shapley details in Section 4.1 and Equation (4) edge cases

We added these reproducibility details to Sections 3.2 and 4.1 and Appendix A.1:

  • Multi-shot permutation protocol and overflow handling (Section 4.1 and Appendix A.1): For each of the 200 targets, we sample 20 random permutations of its 15 incoming neighbors, append demonstrations in permutation order, and average each neighbor's marginal log-likelihood gain across the 20 permutations. On HelpSteer2, all full 15-neighbor prompts fit within a 24,000-token limit without truncation. On HH-RLHF (Appendix A.1), if a coalition exceeds the 8,192-token context window, the scorer keeps the BOS token and left-truncates older prefix tokens so the newly added demonstration and target query at the end stay intact.

  • Aggregation into unique demonstrations and bootstrap CIs (Section 4.1): We report bootstrap 95% confidence intervals on the per-target Spearman rank correlations across all 200 targets (median ρ=0.69\rho = 0.69 [95% CI 0.63, 0.73], mean ρ=0.66\rho = 0.66 [0.63, 0.69]), and clarify that the 1,960 demonstrations in Table 1 correspond to the unique neighbor nodes across the 3,000 directed edges, with repeated appearances averaged per demonstration node.

  • Equation (4) edge cases (Section 3.2): We specify that neighborhood standardization sets the advantage to zero whenever neighborhood variance is zero and omits any neighbor with zero outgoing edges (∣Nout∣=0|\mathcal{N}_{\text{out}}| = 0) from reference statistics.

AuthorsOpenReview ↗

[3/3] Response to Reviewer pw89

Q4: Matched selection budget and overall audit yield for zero-shot likelihood and margin filters

We agree that reporting only Regime I recall in the original Table 3 conditioned the comparison on our method's distinctive slice rather than overall audit yield. In that table, we used a matched budget of 8,500 records (top 5.3% of the corpus, matching our ≈2σ\approx 2\sigma AdveAdv_e pool) for both zero-shot filters: the top 8,500 records ranked by Zero-Shot Likelihood and Zero-Shot Margin include only 3 and 7 of the 968 Regime I errors confirmed in our AdveAdv_e left tail, missing 99.7% and 99.3%.

To evaluate overall audit yield without conditioning on our pool, we also ran full Stage-2 arbitration directly on all 8,500 candidates selected by Zero-Shot Likelihood and Zero-Shot Margin (Section 4.3 and Appendix A.2, Table 4). Because our original Stage-1 score Adve(ni)Adv_e(n_i) (41.4% overall confirmed) depends only on the mean neighbor shift e(ni)e(n_i) (Eq. 2), it flags pairs whose label flips under neighbor context, but misses obvious zero-shot outliers whose low likelihood stays unchanged under neighbors (e(ni)≈0e(n_i) \approx 0). To capture both zero-shot outliers and neighborhood contradictions, we updated the Stage-1 rule in Section 3.3 to combine s0(ni)s_0(n_i) and e(ni)e(n_i) via Advcomb(ni)=z(s0(ni)+w⋅e(ni))Adv_{\text{comb}}(n_i) = z(s_0(n_i) + w \cdot e(n_i)) (Metric-Comb at w=0.75w = 0.75, where every w>0w > 0 improves over w=0w = 0 across w=0.25,0.50,0.75,1.00w = 0.25, 0.50, 0.75, 1.00 in Appendix Table 4):

  • With the unaligned base scorer (Qwen3.5-9B-Base): Because the base model's zero-shot prior s0(ni)s_0(n_i) is not yet calibrated to HHH preferences (50.1% and 50.9% confirmed under paired twin arbitration), conditioning on the 15 nearest neighbors draws a local preference signal from the corpus. Adding our in-context log-likelihood shift e(ni)e(n_i) to s0(ni)s_0(n_i) via Metric-Comb lifts confirmed contradictions to 53.8%, outperforming both Zero-Shot Likelihood and Zero-Shot Margin (Fisher p=1.2×10−4p = 1.2 \times 10^{-4} over Zero-Shot Margin). Thus, in-context conditioning already improves over zero-shot filtering even on an unaligned base scorer.

  • With the post-trained scorer (Qwen3.5-9B): When general HHH preferences are already encoded in s0(ni)s_0(n_i), Zero-Shot Likelihood and Zero-Shot Margin reach 56.5% and 59.1% confirmed under the same paired twin arbitration. Combining s0(ni)s_0(n_i) with e(ni)e(n_i) via Metric-Comb reaches 62.4% confirmed while capturing three times as many negative-shift contradictions (Adve<median⁡Adv_e < \operatorname{median} within each method's 8,500-pair budget: 1,452 vs. 390 and 494). Thus, even when a post-trained scorer makes zero-shot filtering strong on obvious single-record outliers, in-context conditioning adds complementary recall on local convention clashes that look plausible in isolation.

Reviewer_dhj6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a training-free way to find label errors in alignment data. For each sample, it prepends each of its 15 nearest neighbors as a one-shot example and measures how the log-likelihood of the sample's label changes. Samples whose scores fall in the tails are paired with their most contradicting neighbor, and an LLM judges whether one of the two labels is wrong. On HelpSteer2 and HH-RLHF, the flagged samples are judged wrong 1.7–4.5 times as often as random ones, and DPO training on the repaired HH-RLHF data raises reward accuracy by about 0.5% on the full test split and about 3.5% on the repaired test errors.

Critical strengths
  1. The research problem is useful. Preference datasets and benchmarks contain wrong labels, and checking every sample with humans or a strong LLM is expensive.
  2. The motivation is clear. A wrong label can look normal on its own and only stand out next to similar samples. Fig. 2 shows such samples in the left tail of the score, where the zero-shot likelihood is close to the dataset average.
  3. The method is cheap and easy to use. It needs no training, scores the 160,800 HH-RLHF pairs in 9.7 A100 GPU-hours, and sends only 5.3% of the samples to the LLM (Table 3).
Critical weaknesses
  1. All confirmed errors come from LLM judgments, and the LLMs often disagree. There is no human re-annotation (Sec. 5), so a precision such as 32.4% only means that Gemini thinks the label is wrong. On the test split, any two of the three LLM families agree on only 60–68% of pairs (Sec. 4.4). Since each LLM flags only 25–38% of pairs (Appendix B), random answers at these rates would already agree on about 55%. The same arbitrator also changes 9–20% of its own verdicts when run again.

  2. The support from aligned models is partly circular. Aligned models disagree with the human label on 73–83% of confirmed errors (Sec. 4.4). But the samples are selected by a model-likelihood score, so the models already disagree on 61–70% of the flagged pool before arbitration (Table 11). Even the samples cleared by the arbitrator reach 53–61%, against about 37% on the full test split. Much of the gap therefore comes from the selection itself.

  3. The baselines are weak, and the strongest one beats the proposed score. The per-sample DPO loss after 600 steps ranks the confirmed contradictions better (AUROC 0.706 vs. 0.657) at a similar cost. Zero-shot filters are said to miss over 99% of Regime I, which is expected since Regime I is where the zero-shot likelihood looks normal. Data Shapley, TracIn and the LLM judge are compared only by projected cost, and the reward-model ensemble for noisy HH-RLHF pairs [1] is not run.

  4. Prior work on finding label errors from neighbor consistency is not discussed. Earlier work detects corrupted labels from nearby example labels [2], and recent work finds noisy in-context examples by ranking perplexity among semantic neighbors [3]. Demonstrations have also been valued with Shapley values [4]. The new contributions are the directed graph, the twin pairing and the DPO analyses.

  5. The Shapley validation does not cover the score used on HH-RLHF. Adv_e averages how much i's neighbors raise the likelihood of i's label (Sec. 3.2), so it is a neighbor consistency score rather than i's Shapley value. Sec. 4.1 validates only Adv_δ against Shapley, and Adv_δ is no better than random on HH-RLHF (Sec. 4.3). The paper also uses Adv_δ on HelpSteer2 and Adv_e on HH-RLHF without a rule for choosing between them.

Questions for authors
  1. Could you have human annotators re-label a small subset, for example, 150–200 pairs drawn from the flagged, cleared and random pools? Please report the precision against these labels and the agreement between annotators.

[1] Secrets of RLHF in Large Language Models Part II: Reward Modeling, arXiv 2024.

[2] Detecting Corrupted Labels Without Training a Model to Predict, ICML 2022.

[3] On the Noise Robustness of In-Context Learning for Text Generation, NeurIPS 2024.

[4] DemoShapley: Valuation of Demonstrations for In-Context Learning, arXiv 2024.

4 comments
AuthorsOpenReview ↗

[1/3] Response to Reviewer dhj6

We thank Reviewer dhj6 for the constructive review and for noting that our problem formulation is practical, the Regime I motivation is clear, and the two-stage pipeline is lightweight and easy to use. Below we address each of your five points and your question, with all corresponding additions highlighted in blue in the revised PDF.

W1: Reliance on LLM judgments and inter-arbitrator agreement

We appreciate this question and expanded Section 4.4 and Appendix B to compare the three LLM arbitrators (Gemini 3.1 Pro, Llama 4 Maverick, and Claude Opus 4.8) across the 448 flagged test pairs. Because our prompt tells the arbitrator not to flag subjective toss-ups, the three models differ on whether a borderline pair counts as a contradiction (pairwise Cohen's κ=0.14-0.28\kappa = 0.14\text{-}0.28, Fleiss' κ=0.21\kappa = 0.21). However, whenever two models both flag a pair, they agree on which of the two records is wrong 86.2%–98.3% of the time, so taking the majority (≥2/3\ge 2/3, n=118n = 118) or unanimous (3/33/3, n=38n = 38) vote filters out single-model borderline calls. Across repeat runs, verdict agreement is 90.9% on HH-RLHF and 80.0% on HelpSteer2 (where open-ended 0–4 helpfulness grading has more borderline cases).

We also updated our terminology throughout the manuscript to refer to flagged items as arbitrator-confirmed or model-relative preference inconsistencies to be more accurate. In addition, we conducted a blinded human re-annotation study on 100 HH-RLHF test pairs (50 Confirmed, 25 Cleared, and 25 Random) without showing the recorded label (Q1 below and Appendix Table 13): strict human majorities confirm defects on 66.0% of single-arbitrator confirmed pairs and 88.9% of unanimous pairs, while upholding the recorded benchmark label on only 10.0% and 0% (versus 36.0%–40.0% on controls). Thus, while borderline HH-RLHF pairs carry genuine ambiguity for both humans and LLMs, our flagged pairs (especially under multi-arbitrator consensus) are predominantly genuine defects or ambiguous ties rather than valid preferences.

W2: Circularity of aligned-model disagreement on the likelihood-selected pool

You raise a fair point: comparing 73%–83% directly against the 37% full-split rate reflects the combined end-to-end effect of both stages of our pipeline, whereas citing that gap alone overstates the isolated effect of Stage-2 arbitration (and, for Qwen evaluators, shares priors with the Qwen3.5-9B Stage-1 scorer). We updated Section 4.4 to report the full two-stage breakdown in the main text and to frame model–arbitrator agreement as cross-model consistency rather than standalone proof of ground-truth error:

  • Separating Stage 1 and Stage 2: Stage-1 graph scoring narrows the test split from 35%–39% disagreement to a 450-pair candidate pool at 61%–70%. Holding that Stage-1 pool fixed, Stage-2 arbitration by external judges (Gemini, Llama, and Claude) separates cleared pairs (53%–61%) from confirmed errors (73%–83% single-arbitrator, 79%–91% majority, and 87%–95% unanimous).

  • Cross-family check (non-Qwen evaluator): On Gemma3-12B, an unrelated model family not used in Stage-1 selection, both stages contribute: Stage 1 lifts disagreement from 38.6% to 61.3%, and Stage 2 separates same-pool cleared pairs (53.1%) from confirmed errors (72.8% single, 78.8% majority, and 86.8% unanimous, Appendix Table 12).

AuthorsOpenReview ↗

[2/3] Response to Reviewer dhj6

W3: Baselines in Table 3, zero-shot filters, and reward-model ensemble

Thank you for these suggestions. We expanded Section 4.3 and Appendix A.2 (Tables 3 and 4) to evaluate each baseline directly:

  • Zero-shot filters at a matched budget and updated Stage-1 ranking rule: We ran full Stage-2 arbitration across the 8,500-record budget on HH-RLHF for Zero-Shot Likelihood and Zero-Shot Margin (Section 4.3 and Appendix Table 4). Because our original Stage-1 score Adve(ni)Adv_e(n_i) (41.4% overall confirmed) depends only on the neighbor shift e(ni)=1k∑jΔLLi,je(n_i) = \frac{1}{k}\sum_j \Delta LL_{i,j} (Eq. 2), it flags pairs whose label flips under neighbor context, but misses obvious zero-shot outliers whose low likelihood stays unchanged under neighbors (e(ni)≈0e(n_i) \approx 0). To capture both zero-shot outliers and neighborhood contradictions, we updated the Stage-1 rule in Section 3.3 to combine s0(ni)s_0(n_i) and e(ni)e(n_i) via Advcomb(ni)=z(s0(ni)+w⋅e(ni))Adv_{\text{comb}}(n_i) = z(s_0(n_i) + w \cdot e(n_i)) (Metric-Comb at w=0.75w = 0.75, where every w>0w > 0 improves over w=0w = 0 across w=0.25,0.50,0.75,1.00w = 0.25, 0.50, 0.75, 1.00 in Appendix Table 4):

    • With the unaligned base scorer (Qwen3.5-9B-Base, our DPO starting checkpoint): Because the base model's zero-shot prior s0(ni)s_0(n_i) is not yet calibrated to HHH preferences (50.1% and 50.9% confirmed under paired twin arbitration), conditioning on the 15 nearest neighbors draws a local preference signal from the corpus. Adding our in-context log-likelihood shift e(ni)e(n_i) to s0(ni)s_0(n_i) via Metric-Comb lifts confirmed contradictions to 53.8%, outperforming both Zero-Shot Likelihood and Zero-Shot Margin. Thus, in-context conditioning already improves over zero-shot filtering even on an unaligned base scorer.

    • With the post-trained scorer (Qwen3.5-9B): When general HHH preferences are already encoded in s0(ni)s_0(n_i), Zero-Shot Likelihood and Zero-Shot Margin reach 56.5% and 59.1% confirmed under the same paired twin arbitration. Combining s0(ni)s_0(n_i) with e(ni)e(n_i) via Metric-Comb reaches 62.4% confirmed while recovering 3×3\times as many Regime I contradictions. Thus, even when a post-trained scorer makes zero-shot filtering strong on obvious single-record outliers, in-context conditioning adds complementary recall on local convention clashes that look plausible in isolation.

  • Trained reward-model ensemble (Wang et al., 2024): We have now added a three-seed trained DPO reward-model ensemble to our comparison in Appendix A.2 (Table 3): across the 448 arbitrated test pairs, our training-free Metric-Comb achieves comparable error-detection AUROC to the three-model trained DPO ensemble (0.751 vs. 0.731 on single-arbitrator errors and 0.748 vs. 0.756 on majority-confirmed errors), whereas the ensemble requires three full DPO training runs before auditing can begin.

  • Step-600 DPO loss: High per-sample DPO loss does not distinguish corrupted labels from clean, hard-to-learn records, which is why we use step-600 DPO loss in Section 4.3 as an in-training diagnostic reference (or complementary signal) rather than a standalone auditing method. Moreover, on the Table 3 benchmark, our revised Stage-1 score (Metric-Comb) reaches AUROC 0.779, outperforming step-600 DPO loss (0.706) without requiring a training run.

  • Data Shapley, TracIn, and standalone LLM judge: At a matched call budget, Stage-1 filtering lifts confirmed contradictions over an unguided LLM judge from 5.7% to 25.7% on HelpSteer2 (Section 4.2) and anchor-fault rates from 14.9% to 32.4% on HH-RLHF (Section 4.3). Full-corpus Data Shapley and multi-checkpoint TracIn require intractable retraining or full-dataset gradient storage at HH-RLHF scale, so we instead validate against multi-shot Shapley (Section 4.1) and exact LoRA gradients (Section 4.3) on subsamples.

W4: Missing prior work on neighbor consistency and in-context valuation

Thank you for these references. We have added all three to Section 2 (Related Work).

W5: Shapley validation of incoming scores on HH-RLHF and rule for choosing between scores

We appreciate this feedback and added both analyses to the revised manuscript: (continued in next comment)

AuthorsOpenReview ↗

[3/3] Response to Reviewer dhj6

(Continued) W5:

  • Clarifying incoming scores, updated title, and multi-shot check on HH-RLHF (Sections 3.2 and 4.1): We agree with your point: incoming e(ni)e(n_i) (normalized as AdveAdv_e) averages the one-shot shifts received from nin_i's neighbors and is therefore a neighborhood label-consistency score rather than node ii's Shapley value. Because our original title overstated the Shapley framing for the incoming score, we changed the paper's title to "In-Context Influence Graphs: Scalable Data Auditing for LLM Alignment and Evaluation" to avoid confusion and clarified this distinction in Section 3.2. On 200 HH-RLHF evaluation targets (Section 4.1), we also verified that e(ni)e(n_i) strongly tracks the rank of the 15-neighbor multi-shot likelihood shift (ρ=0.70\rho = 0.70, 95% CI [0.62, 0.77]), and that the worst one-shot neighbor selected for Stage-2 arbitration has a much more negative multi-shot Shapley value than the remaining neighbors (−0.23-0.23 vs. −0.09-0.09 nats, p=1.3×10−9p = 1.3 \times 10^{-9}).

  • Rule for choosing between outgoing and incoming scores (Sections 3.2 and 3.3): Outgoing influence (AdvδAdv_\delta) measures demonstration utility, so we use it to select strong exemplars or to audit corpora with explicit quality ratings where yiy_i is an open-ended response rather than a label token. On HelpSteer2 (Section 4.2), for example, the 4/44/4 rating is external metadata asserting that the response is exemplary, so placing the response in-context via AdvδAdv_\delta tests whether it helps or degrades predictions on same-grade peers. Conversely, incoming influence (e(ni)e(n_i) / AdveAdv_e) scores yiy_i in the target slot to measure neighborhood label consistency, so we use it whenever yiy_i directly encodes the preference choice without an external quality grade, as in HH-RLHF (Sections 4.3–4.5).

Q1: Blinded human re-annotation of stratified pairs

We ran a blinded human re-annotation study on 100 stratified HH-RLHF test pairs (50 Gemini-confirmed, 25 Stage-1 cleared, and 25 random, Section 4.4 and Appendix Table 13). Raters saw randomized Response A/B order without seeing the recorded benchmark label or pool origin and selected A better, B better, Tie (both acceptable), or Both unacceptable under the HHH rubric (314 ratings from 17 raters, ≥3\ge 3 per pair):

Pool (nn pairs)Defect Majority (Inv or Both bad)Tie / Split (No >50% side)Orig. Validated (Orig >50%)Decisive Flip (Inv > Orig)
Random control (25)56.0% (14/25)4.0% (1/25)40.0% (10/25)47.6% (10/21)
Stage-1 Cleared (25)56.0% (14/25)8.0% (2/25)36.0% (9/25)47.6% (10/21)
Confirmed: Single LLM (50)66.0% (33/50)24.0% (12/50)10.0% (5/50)67.4% (31/46)
↪\hookrightarrow Majority ≥2/3\ge 2/3 LLMs (25)76.0% (19/25)20.0% (5/25)4.0% (1/25)66.7% (16/24)
↪\hookrightarrow Unanimous 3/33/3 LLMs (9)88.9% (8/9)11.1% (1/9)0.0% (0/9)75.0% (6/8)
↪\hookrightarrow Repaired flips (31)71.0% (22/31)22.6% (7/31)6.5% (2/31)78.6% (22/28)

Precision and control baselines: Human defect precision rises monotonically from 66.0% on single-LLM pairs to 76.0% and 88.9% under multi-arbitrator consensus, and among Repaired flips with unequal directional votes (Inv ≠\neq Orig), more raters prefer our flipped label than the recorded one on 78.6% of pairs versus 47.6% on controls (Fisher p=0.035p = 0.035). Raters also mark 56.0% of uniform Random pairs as defective, consistent with the ~63% researcher–crowdworker agreement reported on HH-RLHF by Bai et al. (2022) (p=0.45p = 0.45 on single-LLM binary defects, p≥0.11p \ge 0.11 on the smaller consensus tiers). Because 12 of the 17 remaining Confirmed pairs are acceptable ties or split votes rather than validated preferences, human majority validation of the recorded benchmark label drops significantly from 36.0%–40.0% on controls to 10.0% on Confirmed pairs (p≤0.011p \le 0.011) and 0% on unanimous pairs.

Inter-annotator agreement: Across all 342 rater pairs, human agreement is moderate across all four options (52.3% pairwise, Krippendorff's α=0.31\alpha = 0.31) and on binary defects (68.7%, α=0.36\alpha = 0.36), largely because raters differ on when to use Tie or Both unacceptable. When two raters both pick A or B, pairwise agreement is 71.4% overall (α=0.42\alpha = 0.42) and 79.3% on Confirmed pairs (α=0.57\alpha = 0.57), while staying near chance at 55.7% on the more ambiguous Stage-1 cleared pairs (α=0.13\alpha = 0.13).

Overall, blinded human adjudication shows that raw HH-RLHF carries substantial baseline label noise and that our flagged items (especially under multi-arbitrator consensus) are predominantly genuine defects or ambiguous ties rather than valid preferences.

Thank you for the detailed experiments. My main concern remains whether the method identifies genuine preference-label errors more reliably than simpler filters. The human study provides some support, but the improvement over random sampling is modest, and the comparison with zero-shot filtering still relies on LLM judgments. I will therefore maintain my score.

Reviewer_61UtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a training-free way to detect mislabeled or low quality records for LLM alignment purposes, by taking the k most similar ones (according to embedding sim.), and measuring how much a neighbour in the context changes the likelihood of the considered item's response (and vice versa). After filtering, candidate pairs are judged by an LLM. Results on 2 well-known benchmarks, there is a gain in hit rate for the records that were flagged by the detection procedure. After an intrinsic evaluation of the proposed filtering strategy, a downstream alignment training run on cleaned vs original pairs seems to show positive effects.

Critical strengths

The procedure seems well-engineered, and could turn out useful in practice. The authors provide results on 2 existing benchmarks, both intrinsically (on the detection of low-quality training items), as well as extrinsically (impact on alignment of filtering in alignment training data).

Critical weaknesses

I think in its current form, the manuscript is not ready for publication at ICLR, and below are what I consider the key issues.

  • Results seem overstated in the main body (critical)

    • Whereas table 2 in the main body reports gains on the order of 0.5 - 0.6 percentage points, this seems to be a result on a single seed. Indeed, in Table 14 at the very end of the appendix, the difference between repaired and uncleaned accuracy, averaged over 5 seeds, seems to be only 0.15 percentage points, similar to the standard deviation per arm (0.18-0.15%). That means the difference would not be significant. In addition, these are not independent seed results; only the last 254 steps are separate, after 1000 steps in common. That means the true independent arm standard deviation is likely larger than the reported value, leading to even larger p values. Maybe I did not understand the entire experimental setup, but to me this puts the merit of the method in question.
  • Framing as a Shapley method

    • As far as I am aware, classical Shapley requires features to be independent. However, in the current context, examples can be strongly correlated (e.g. near duplicates); how are these cases handled here, and what does it mean for the Shapley interpretation of DeltaLL?
    • Even though the title starts with 'First-order in-context shapley', the method is not explicitly introduced as a Shapley value, that remains somewhat vague (and therefore issues like the one above are not discussed).
    • In general, there are no principled derivations or proofs. That is not a problem by itself, but the title alone hints at a formal method introduction, and the paper does not meet these expectations.
  • Strongly related work not referenced

    • Gao et al., "On the Noise Robustness of In-Context Learning for Text Generation", NeurIPS 2024 seems similar, in using ppl in kNN neighbourhoods to detect noisy examples, although for the application of in-context learning instead of alignment.
    • Xie et al., "DemoShapley: Valuation of Demonstrations for In-Context Learning", BigData 2025, also in-context learning,
  • Clarity issues

    • Part of the abstract is hard to understand without having read the paper. Especially the typical LLM-style listing of results in-line in the text, reduces readability.
    • The introduction is written clearly, apart for the part of the contributions, which is very hard to understand at that point in the paper. The contributions should be described in a clear way, even if mainly qualitatively, now they require concepts introduced much later (symbols like Adve_e, Adv_delta, vague concepts like LLM arbitration etc). These parts are not humanized enough.
Questions for authors

Please provide your feedback on the critical points listed above.
In particular, what are the mean accuracies for actually independent seed runs of both arms? Please report on significance, e.g., with the McNemar test. These are the results to be reported in the key tables in of the main body.

3 comments
AuthorsOpenReview ↗

[1/2] Response to Reviewer 61Ut

We thank Reviewer 61Ut for the thoughtful review and for highlighting that our procedure is well-engineered and evaluated both intrinsically and extrinsically. Below we address each of your points and questions, with all corresponding additions highlighted in blue in the revised PDF.

W1: Results seem overstated in the main body

Thank you for commenting on this and for requesting multi-seed evaluation. In the original submission, Table 2 reported single-seed from-scratch runs while Table 14 (now Table 16) resumed five seeds from a shared step-1,000 checkpoint.

To evaluate full end-to-end seed variance directly as you suggested, we ran three independent full from-scratch DPO seeds per arm with the same training setup, alongside size-matched random-deletion controls (reported in the updated Table 2 in Section 4.5 and Appendix Table 15). We would like to emphasize three points regarding the magnitude of the downstream gains:

  • Modest full-split gains (Sections 4.5 and 5): We agree that the accuracy gain on the full evaluation split is modest (~0.2–0.3 pp across seeds, within seed-level variance), as our arbitrated arms modify only 2.4% of the training corpus. We updated the Abstract, Introduction, Section 4.5, and Section 5 (Limitations) accordingly to note that full-split differences lie within seed variance and to focus our downstream retraining claims on the corrected test inversions. To verify this sensitivity ceiling directly, we ran a controlled 10%-corruption DPO experiment (16,053 flipped training pairs out of 160,490) comparing the corrupted floor, size-matched random deletion (16,053 pairs), and an exact ground-truth oracle that removes all 16,053 verified flips:
Training Arm on 10%-Corrupted HH-RLHF (16,053 Flips)Full Raw (8,535)Repaired (8,486)
Uncorrupted Baseline (Uncleaned, 0% flips)68.45%69.18%
Corrupted Floor (10% flips kept)67.43%68.04%
Random Deletion (16,053 pairs removed)67.44% (+0.01%)68.15% (+0.11%)
Exact Oracle (all 16,053 verified flips removed)69.02% (+1.59%)69.77% (+1.73%)

Even when an exact ground-truth oracle removes 10% verified label corruptions (roughly 4×4\times the volume modified by our arbitrated audit), Full Raw accuracy shifts by only +1.59% over the corrupted floor and +1.73% on Repaired Eval, whereas random deletion of 16,053 pairs stays flat (+0.01% and +0.11%). Thus, when our arbitrated audit modifies only 2.4% of records, a ~0.2%–0.3% full-split shift is expected.

  • Role of downstream DPO retraining: Our primary focus is lightweight, training-free auditing of training corpora and evaluation benchmarks. Downstream DPO accuracy gains depend on factors beyond audit precision, including the small fraction of records modified and label noise in the raw evaluation split.

  • Larger gains on cleaned evaluation splits and corrected inversions: Because raw HH-RLHF test labels themselves contain preference inconsistencies, evaluating on the cleaned test splits (Sanitized and Repaired) yields clearer gains than on Full Raw. Most directly, on the corrected test inversions, every curated arm improves over Uncleaned (with all 12/12 runs beating their matched uncleaned seed on majority inversions), whereas random deletion lowers accuracy.

W2: Framing as a Shapley method

Thank you for this helpful perspective on how the title and framing set reader expectations. Our goal in Section 3 is to use the local in-context cooperative game as motivation for a tractable, first-order directed influence graph rather than to develop new theoretical Shapley bounds. To make the paper's practical focus immediately clear, we have updated the title to "In-Context Influence Graphs: Scalable Data Auditing for LLM Alignment and Evaluation" and written out the local cooperative game and its singleton truncation explicitly in Section 3.2.

Regarding correlated or near-duplicate examples: the cooperative-game Shapley definition over a set of demonstrations requires no independence assumption. Instead, correlation affects how credit is shared across coalitions: in multi-shot subsets, redundant or near-duplicate neighbors act as substitutes and split their marginal contribution across copies, whereas our first-order probe evaluates each neighbor individually against the zero-shot baseline so that identical neighbors receive identical ΔLLi,j\Delta LL_{i,j} scores without dilution (Section 3.2).

W3: Strongly related work not referenced

Thank you for pointing us to these references. We have added Gao et al. (NeurIPS 2024) and Xie et al. (DemoShapley, BigData 2025) to Section 2 (Related Work).

AuthorsOpenReview ↗

[2/2] Response to Reviewer 61Ut

W4: Clarity issues in the Abstract and Introduction

Thank you for this suggestion. We revised both the Abstract and the Introduction contribution bullets to present our method and findings in clear, qualitative prose, and removed unnecessary numbers. Before defining mathematical notation in Section 3, we also describe the three record-level signals in plain terms: outgoing influence (how a record affects predictions on its neighbors), incoming influence (whether a record's label agrees with its semantic neighborhood), and a uniqueness score that protects rare prompts from deletion.

Q1: Mean accuracies and significance for independent from-scratch seed runs

As described under W1, Table 2 in the main body (Section 4.5, full breakdown in Appendix Table 15) now reports mean ± SD across three independent from-scratch seeds per arm with the same training setup. While full-split accuracy differences across from-scratch seeds are modest and within seed-level variance, every curated arm outperforms Uncleaned on both inversion splits, with all 12/12 runs beating their matched uncleaned seed on majority-confirmed inversions and statistically significant gains in accuracy and preference margin:

  • Full splits (Repaired vs. Uncleaned): 68.77 ± 0.26% vs. 68.66 ± 0.19% on Full Raw, and 69.59 ± 0.28% vs. 69.38 ± 0.18% on Repaired Eval (69.68 ± 0.19% for Sanitized). Neither paired seed tt-tests (p=p = 0.16-0.41) nor McNemar tests (p≥0.20p \ge 0.20 for Repaired and Sanitized, Appendix B) reach significance on these full splits.
  • Corrected test inversions (n=142n = 142): 70.19 ± 0.41% for Repaired (up to 74.18 ± 1.08% across curated arms) vs. 68.08 ± 1.63% for Uncleaned (Metric-Comb accuracy p=0.027p = 0.027, Repaired margin p=0.006p = 0.006).
  • Majority-confirmed inversions (n=97n = 97): 76.98 ± 1.57% for Repaired (up to 78.35 ± 1.03% across curated arms) vs. 73.54 ± 2.59% for Uncleaned (Metric-only accuracy p=0.010p = 0.010, Repaired margin p=0.028p = 0.028).

I thank the authors for their efforts and the extensive rebuttal. I have to say that there is an actual improvement in the presentation (abstract, intro), and the new title is no longer misleading. Also, the additional results are much appreciated, although I think they should have been done before initial submission, to give the authors a more realistic view on the method's effectiveness, rather than merely to satisfy one reviewer. Indeed, it seems that the original presentation (single seed) was a bit optimistic, and the independent-seed experiments show that the actual improvements are very small (and actually not significant). The authors do acknowledge that transparently. Still, I'm now even less convinced about the practical utility of the proposed empirical method, and the added value compared to zero-shot filtering (or even simple filtering without LLM use). For that reason, I stand by my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.