acceptodds
Under review as a conference paper at ICLR 2027

Shapley-based Data Valuation for LLM Alignment via Sequential Preference Optimization

Abstract

Data valuation is a natural framework for understanding which data sources matter most when aligning a Large Language Model (LLM) from multiple sources. The standard game-theoretic approach treats each source, or equivalently each preference dataset, as a player in a cooperative game and assigns it a contribution score through the Shapley value. In practice, however, Shapley-based valuation is computationally prohibitive because it requires aligning a separate model for every possible coalition of sources, i.e., an exponential number of alignments. We address this challenge for Direct Alignment Algorithms (DAAs), including IPO, which learn through log-policy ratios with respect to a reference policy. We show that, when a model is aligned sequentially source by source, exact optimization makes each stage contribute additively to the log-probability of a full response, up to a prompt-dependent normalization constant. This allows the log-probability assigned by any coalition to a fixed response to be reconstructed from the base policy and the policies trained on each source individually. This reduces the alignment cost of Shapley-based valuation from exponential to linear, since only one model per source needs to be trained to evaluate coalition scores. We test whether this theoretical property remains approximately valid under finite training across several base models and real-world data sources. We finally compute the Shapley values of these sources under multiple reward models, showing how their estimated contributions vary across evaluation criteria.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Data valuation, Shapley value, LLM alignement

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ez31OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies source-level Shapley valuation for preference-based LLM alignment. It defines coalition models through sequential preference optimization, using each stage's policy as the next stage's initialization and reference. Under an attained and identifiable population optimum over the score space, each source induces a reference-independent relative log-probability increment. Coalition complete-response log-probabilities can then be reconstructed, up to a prompt-dependent constant, by adding singleton log-policy ratios to the base model's scores. For shift-invariant evaluation protocols, this reduces the number of trained models from one per coalition to one per source; exact Shapley summation still requires all coalition utilities. Experiments examine order sensitivity and reconstruction discrepancies for nine UltraFeedback provenance sources, using IPO on four base models and supplementary DPO experiments on two. The application computes exact Shapley sums for a reconstructed game that selects among 16 fixed base-model candidates and evaluates selections with three reward models.

Critical strengths

S1. A precise way to remove coalition-specific alignment for a defined valuation game. Lemma 3.1 and Propositions 3.1-3.2 show how reference-independent score increments produce an order-invariant coalition policy and how singleton policies encode its complete-response scores. Section 3.4 identifies the class of shift-invariant utilities for which the unknown normalization is irrelevant. Under these assumptions, the main benefit is avoiding coalition-specific retraining while preserving the specified utility. Exact Shapley computation still requires exhaustive utility enumeration.

S2. The empirical study tests the structural predictions and includes useful controls. Figure 1 compares both order changes and reconstruction against repeated-training variability across four models. Appendix D.1 repeats the tests with DPO, and Appendix D.2 distinguishes the sequential procedure from pooled IPO, reporting 1.3%-3.0% disagreement on held-out preference-pair rankings for SmolLM (Table 1). These checks provide preliminary evidence beyond the idealized derivation and help delineate what the proposed alignment procedure measures.

Critical weaknesses
  • W1. The reported validation does not establish fidelity for the actual utilities and source values. My main concern is a mismatch between the validation metric and the valuation application. Sections 4.2-4.5 measure mean absolute errors of the original held-out preferred/rejected log-probability gaps and report discrepancies within twice a three-seed retraining reference. Section 5 and Appendix C.6 apply an argmax to 16 newly generated candidates and evaluate the selected response's reward. Agreement on the original pair does not constrain the other candidates' rankings, and argmax utilities can change sharply near a decision boundary. Neither Figure 1 nor the DPO counterpart reports selection agreement or reward-utility error on these candidate pools. The tests also cover only three nested coalitions, while Figure 2 aggregates marginal utilities over all 512 coalitions without reported valuation uncertainty. The application therefore lacks direct validation of finite-training source-value fidelity. Evaluating the existing coalition models on the same candidate pools, together with an exact small-source Shapley comparison and uncertainty estimates, could substantially alleviate this concern.

  • W2. The practical scope of the identifiable population model needs clarification. Assumption 3.1 requires each source's minimizers to agree on every response's relative score for every prompt. This is substantially stronger than exact optimization. For example, with one prompt, three responses, beta = 1, and only the comparison a preferred to b, the score vectors (1/4, -1/4, 0) and (1/4, -1/4, 1) both achieve zero IPO loss but induce different probabilities for the unobserved response. Exact optimization alone therefore does not identify a source-specific update. The highest/lowest completion construction in Appendix C.1 does not establish the necessary population comparison coverage; likewise, a fixed neural policy class does not automatically realize the exponential tilts used in Appendix B. The acknowledged idealization would be more informative with sufficient data-support conditions and an explanation of how unconstrained directions are selected in practice. This concern limits the theorem's explanatory scope; it does not contradict the propositions under their assumptions.

Questions for authors
  • Q1. Validation of candidate selection (W1). For the already trained coalitions and orders, what are the response-selection agreement and standardized reward-utility discrepancies between reconstruction and explicit sequential models on exactly the 16-candidate pools used in Section 5? Please compare these discrepancies with same-procedure seed variability.
  • Q2. Validation of source values (W1). In a three-source subgame with all eight coalition utilities evaluated explicitly under a specified ordering/seed protocol, how close are the reconstructed and explicit Shapley values? Please report value errors and sign/rank stability, with uncertainty from training and evaluation sampling.
  • Q3. Clarification of the assumptions (W2). What sufficient conditions on each source's prompt support and response-comparison distribution ensure Assumption 3.1, including finite attainment for DPO? Does the implementation impose any reference-independent rule on score directions left unconstrained by observed comparisons, and what policy-class realizability is needed for the exponential-tilting argument?
5 comments

We thank the reviewer for their careful reading and constructive questions. We address each point below in the order raised. Throughout our answers, all references to sections, tables, or appendices refer to the paper revision submitted with this rebuttal. A summary of these changes is provided in the general comment titled Paper revision.

Weaknesses [1/2]

W1. The reviewer argues that Section 4 validates the reconstruction on log-probability gaps between two fixed responses, whereas Section 5 relies on a different object, namely the candidate selected among 16 and its reward. Agreement on the original pair does not constrain how the other candidates are ranked, and an argmax can change abruptly near a decision boundary. The reviewer also notes that Section 4 covers only three nested coalitions, and that Figure 2 reports no uncertainty. We thank the reviewer for this precise analysis, and we agree that the validation should cover the quantities used in Section 5. The three checks suggested by the reviewer are well suited to this question. We carried out all of them, and they substantially strengthen the paper's empirical support.

  • (a) Validation on the selections and utilities of Section 5. We evaluated the coalitions of Section 4 on exactly the 16-candidate sets of Section 5. We compared the reconstruction with the explicitly trained coalitions and used two runs that differ only in their seed as a reference. We observe that the reconstruction is as close to the explicit coalitions as a retraining is, both in the selected candidates and in the utilities. The sensitivity of the argmax near decision boundaries therefore affects the reconstruction no more than an independent retraining. We report this experiment in Appendix D.6, and our answer to Q1 gives the numbers.

  • (b) Exact Shapley values on a small set of sources. We explicitly trained all 31 coalitions of five sources. We observe that the reconstructed values recover the signs and the order of the sources, with small errors. We report this experiment in Appendix D.3, and our answer to Q2 gives the numbers.

  • (c) Uncertainty. We report 95% confidence intervals over the evaluation prompts in Appendices D.3 and D.6, and we compare every discrepancy in Appendix D.6 with the variability between runs that differ only in their seed. We also computed 95% confidence intervals for the nine values of Figure 2 in Appendix D.5. We observe that every value that differs significantly from zero keeps its sign when the candidate sets change.

Weaknesses [2/2]

W2. The reviewer makes three points about Assumption 3.1. First, exact optimization does not imply uniqueness, as the example with three responses shows. Second, comparing only the highest and lowest completions of each prompt does not provide the coverage that uniqueness requires. Third, a neural policy class need not realize the updates used in the proofs. The reviewer therefore asks how the directions left unconstrained by the data are selected in practice. We thank the reviewer for these observations, which are correct and helped us clarify the theoretical role of Assumption 3.1. We do not claim that this assumption holds in practice. Its role is to define an idealized regime in which sequential alignment on a coalition provably reduces to singleton alignments. This regime explains why coalition scores should be reconstructible from singleton policies, and it predicts the properties that we then test under finite neural training in Section 4. We now state this role explicitly after Assumption 3.1, and we address the three points of the reviewer in the new Appendix F.3.

  • (a) When uniqueness holds. We prove in Lemma F.1 the exact condition for uniqueness. For IPO and DPO, the minimizers of a source are unique up to a prompt-dependent constant if and only if, at every prompt, the response pairs compared by the source form a connected graph. In the reviewer’s example, the third response is isolated, so uniqueness fails. For the same reason, it fails for our sources, since each source compares a single pair per prompt and the sources have different prompt supports.

  • (b) Why we assumed uniqueness, and why the reconstruction does not need it. We assumed uniqueness so that each coalition has a single exact policy, whichever minimizer each stage returns. The coalition utility is then unambiguously defined, and Proposition 3.1 can state that every exact sequential run returns this policy. However, this is the only role of uniqueness. We prove in Proposition F.1 that the reconstruction remains exact without it, as long as each singleton objective attains its minimum. We fix one minimizer for each source, and consider the run of the sequential procedure in which every stage selects the minimizer of the corresponding singleton. We show that this run is exact for every ordering, that all orderings return the same policy, and that this policy satisfies the reconstruction identity of Proposition 3.2. In the camera-ready version, we will therefore replace Assumption 3.1 by this weaker assumption, and state the results of Section 3 for the coalition policy obtained when each stage applies the update of the corresponding singleton, with uniqueness as a remark.

  • (c) What happens in practice. Our training adds no constraint on the scores that the data leave free, such as those of responses that a source never compares. However, we initialize each stage at its reference policy, as described in Appendix C.4. Each stage therefore starts from a zero update relative to its reference, exactly as the singleton alignment starts from the base policy. This mirrors the selection described in Proposition F.1, although finite training only approximates it. As for realizability, we state the theory over all full-support policies, and we do not assume that a finite neural model realizes each update. Whether finite training behaves like the idealized regime is therefore an empirical question. Section 4 shows that, under finite training, coalition scores remain as close to their reconstruction as two runs that differ only in their seed, and Appendices D.1 and D.6 extend this test to DPO and to the quantities of Section 5.

Questions

Q1. The reviewer asks for the selection agreement and reward-utility discrepancies between the reconstruction and the explicit sequential models on the 16-candidate sets in Section 5, compared with seed variability. This is a relevant test, since it evaluates the reconstruction directly on the objects of Section 5. We measured these quantities and report them in Appendix D.6. We used the coalitions, orders, and seeds from Section 4, along with the candidate sets, reward models, and standardization from Section 5, so no additional training was needed. We observe that the reconstruction selects the same candidate as the explicit coalition on 68.6% to 87.1% of the prompts, compared with 72.3% to 88.7% for two runs that differ only in their seed. The ratio of reward discrepancies, which we build as ρrecon\rho_{\mathrm{recon}} in Section 4, lies between 1.08 and 1.62. The utility errors, between 1.0 ×10−2\times10^{-2} and 3.2 ×10−2\times10^{-2} in reward standard deviations, are of the same order as those between seeds, between 0.5 ×10−2\times10^{-2} and 3.0 ×10−2\times10^{-2}. Both are of the order of the uncertainty due to the finite number of evaluation prompts. Table 7 reports all values.

Q2. The reviewer asks for a subgame in which all coalitions are explicitly trained, with value errors and sign and rank stability. We found this suggestion particularly valuable, and we followed it on five sources rather than the suggested three. We explicitly trained the 31 resulting coalitions for the four base models, computed their Shapley values under the three reward models, and report the comparison in Appendix D.3. We observe that the reconstructed values deviate from the explicit values by 0.5 ×10−2\times10^{-2} to 0.9 ×10−2\times10^{-2} on average, whereas the explicit values range from 0.03 to 0.13. The reconstruction recovers every sign and correctly orders every pair of sources that the explicit game separates with 95% confidence. Its Spearman correlation with the explicit values lies between 0.9 and 1.0, and between 0.6 and 0.8 for Phi-3.5, whose five values are close to each other. Table 2 reports these results with 95% confidence intervals over the evaluation prompts. We trained each coalition of Appendix D.3 with a single seed, so these intervals only account for the finite number of evaluation prompts. We measured training variability at the level of coalition utilities in Appendix D.6, where the errors of the reconstruction are of the same order as those between seeds, as detailed in our answer to Q1.

Q3. The reviewer asks for sufficient conditions for uniqueness and attainment, for the realizability required by the proofs, and for the rule applied to unconstrained directions. These questions helped us make the theory's scope explicit. For uniqueness, we prove in Lemma F.1 a necessary and sufficient condition, namely a connected comparison graph at every prompt. For attainment, IPO needs no condition, since its quadratic loss always has a minimizer, as we now state in Appendix F.3. For DPO, the logistic loss attains its minimum if and only if, within every connected component of the comparison graph of a prompt, no group of responses wins every comparison against the others. This is the classical condition for the existence of the Bradley–Terry estimator. It fails for our sources, since each prompt has a single preferred response. Our DPO results in Appendix D.1 therefore provide evidence under finite training rather than a test of the exact regime. Finally, we discuss realizability and the rule on unconstrained directions in our answer to W2-(c).

If our responses and the revised paper address the reviewer's concerns, we would be grateful if they would consider updating their assessment. We would be happy to clarify any remaining points during the discussion period.

Reviewer_QywwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies source-level data valuation for LLM alignment using a sequential preference-optimization procedure, where the reference policy is updated after training on each source. Under exact population optimization and an identifiability assumption, the authors show that source updates compose additively in complete-response log-probability space. Consequently, coalition scores can be reconstructed from the base policy and one separately aligned model per source, up to a prompt-dependent constant. This reduces the number of required alignments from exponetial to linear.

Critical strengths

The main strength is that under exact and identifiable population optimum, each source induces a reference-independent change in relative response log-probabilities. This makes reconstruction rule from singleton policies and makes coalition-specific alignment unnecessary, and reduces the number of required alignments from exponential to linear.

Critical weaknesses

The main assumption requires each source to identify relative response scores at every prompt. In the unrestricted setting, missing prompt support or disconnected comparison graphs violate this requirement even with exact optimization. To me this leaves the theory’s applicability to heterogeneous sources insufficiently justified.

Questions for authors

What justifies this assumption for your sources? Can an optimizer-selection rule preserve reference-independent updates when it fails?

4 comments

We thank the reviewer for the positive assessment and the clear summary of our contribution. We are glad that the reviewer finds the paper clear and sound. The reviewer’s main concern is that the identifiability required by Assumption 3.1 fails for heterogeneous sources, since sources may miss some prompts or have disconnected comparison graphs. To address this concern, we now make explicit in the revised paper that the reconstruction does not require this identifiability, and we prove this in a new theoretical result. We present this result in our answer to the weakness below, and build on it to answer the two questions. All references to sections, tables, or appendices refer to the paper revision submitted with this rebuttal, which is summarized in the general comment titled Paper revision.

Weaknesses

W1. The reviewer notes that Assumption 3.1 requires each source to identify relative response scores at every prompt. Missing prompt support or disconnected comparison graphs violate this requirement, even under exact optimization, so the applicability of the theory to heterogeneous sources appears insufficiently justified. We thank the reviewer for this correct observation. We first clarify the role that Assumption 3.1 plays in the paper. We do not claim that it holds in practice. It defines an idealized regime in which sequential alignment on a coalition provably reduces to singleton alignments. This regime explains why coalition scores should be reconstructible from singleton policies, and it predicts the properties that we test under finite neural training in Section 4. We now state this role explicitly after Assumption 3.1. Building on the reviewer’s remark, we then studied precisely which part of the assumption the theory needs, in the new Appendix F.3.

  • (a) The identifiability condition fails for heterogeneous sources. We prove in Lemma F.1 that, for IPO and DPO, the minimizers of a source are unique up to a prompt-dependent constant if and only if, at every prompt, the response pairs compared by the source form a connected graph. This confirms the reviewer’s analysis. Uniqueness fails as soon as a source does not cover a prompt, or compares responses that do not connect all candidates. Both situations occur for our sources.

  • (b) The reconstruction does not need it. We assumed uniqueness so that each coalition has a single exact policy, whichever minimizer each stage returns. However, this is the only role of uniqueness. We prove in Proposition F.1 that the reconstruction remains exact without it, as long as each singleton objective attains its minimum. In that case, a natural selection rule preserves reference-independent updates, as we detail in our answer to Q2.

  • (c) The theory therefore applies to heterogeneous sources. The remaining condition, attainment, always holds for IPO, which we use in our main experiments, since its quadratic loss has a minimizer for any comparison data. We also strengthened the empirical support for heterogeneous sources. We explicitly trained all 31 coalitions of five sources for four base models, and compared their Shapley values with the reconstructed ones in Appendix D.3. We observe that the reconstruction recovers every sign and the order of the sources, with an average error of 0.50.5 ×10−2\times10^{-2} to 0.90.9 ×10−2\times10^{-2} in reward standard deviations, whereas the values range from $0.03 $ to 0.130.13. We also evaluated the coalitions of Section 4 on the candidate sets used for valuation in Appendix D.6. We observe that the reconstruction is as close to the explicitly trained coalitions as retraining with another seed.

Questions

Q1. What justifies this assumption for your sources? Uniqueness is a simplifying idealization, which we adopted for two reasons. First, it gives each coalition a single exact policy, so the coalition utility is well defined without specifying which minimizer each stage returns. Second, it turns order invariance into a statement about every exact run of the sequential procedure, keeping the derivation of Section 3 short and isolating its central mechanism, namely that each source induces the same update regardless of its reference. However, the reviewer is right that uniqueness does not hold literally for our sources. Lemma F.1 shows that it fails, since each source compares a single pair of responses per prompt and the sources have different prompt supports. Proposition F.1 shows that this does not affect the reconstruction, which only requires that each singleton objective attains its minimum. This condition always holds for IPO, which we use in our main experiments. Following the reviewer’s remark, we will replace Assumption 3.1 in the camera-ready version with this weaker assumption and state the Section 3 results for the coalition policy obtained with the selection rule described in our answer to Q2. We would welcome the reviewer’s opinion on this change. Assumption 3.1 thus provides an idealized regime that explains the mechanism, and we test its predictions under finite training in Section 4 and in Appendices D.3 and D.6.

Q2. Can an optimizer-selection rule preserve reference-independent updates when it fails? Yes, and we thank the reviewer for this question, which describes exactly the result of Proposition F.1. When uniqueness fails, the objective of a source admits several minimizers, which differ on the scores that the source does not constrain, such as those of responses it never compares. A stage of the sequential procedure could then select a different minimizer depending on its reference, and the update induced by the source would no longer be reference-independent. Proposition F.1 shows that the following rule prevents this. We choose once, for each source, one minimizer fi⋆f_i^\star of its singleton objective, that is, one update of the log-probabilities. Every stage then applies this same update, whatever its reference. We prove that this update is always a minimizer at the corresponding stage, so the resulting run of the sequential procedure is exact. Since each source always applies the same update, the coalition policy is the base policy shifted by the sum of the singleton updates. It is therefore the same for every ordering, and it coincides with the reconstruction of Proposition 3.2. This rule is also natural in practice. We initialize each stage at its reference policy, so each stage starts from a zero update, exactly as the singleton alignment does. Whether finite training follows this rule is an empirical question, and Section 4 and Appendix D.6 show that it does so up to the variability between runs that differ only in their seed.

We thank the reviewer again for their positive assessment and for their questions that led to a clearer theoretical statement. We hope that the new result in Appendix F.3 and the additional experiments address the remaining concern about applicability, and we would be happy to clarify any point during the discussion period.

I thank the authors for their rebuttal. I will keep my positive assessment.

We thank the reviewer for taking the time to read our rebuttal and for maintaining their positive assessment.

Reviewer_aRCkOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an efficient method to compute Shapley values for preference data sources in Direct Alignment Algorithms (DAAs, such as IPO and DPO). Standard Shapley valuation over nn sources requires 2n−12^n - 1 coalition-specific alignment runs. To bypass this exponential cost, the authors define a sequential preference optimization procedure—aligning source by source while updating the reference policy at each stage—and show that under exact, unconstrained population optimization, source updates compose additively in sequence-level log-policy-ratio space (Propositions 3.1–3.2). Consequently, the sequence log-probability of any coalition policy can be reconstructed up to a prompt-dependent constant using only the base policy and nn singleton policies. The authors test order invariance and reconstruction error on four small instruction-tuned models (135M to 3.8B) across n=9n=9 UltraFeedback subsets, and compute reconstructed Shapley values using three external reward models via Best-of-16 reranking.

Critical strengths
  1. Clear exposition and clean observation. Reparameterizing DAAs in log-policy-ratio coordinates makes the reference-policy cancellation transparent, and the distinction between sequence-level score reconstruction and next-token autoregressive generation (Appendix F.2) is explained clearly.
  2. Substantial reduction in training runs. Reducing 2n−12^n - 1 trainings to nn singleton runs (plus teacher-forced scoring) addresses a genuine bottleneck in source-level data attribution.
  3. Careful separation of data splits. Prompts are cleanly split into training, per-stage checkpoint validation, and held-out evaluation sets across four model families and two losses (IPO and DPO).
Critical weaknesses

W1. Overly strong theoretical assumption (Assumption 3.1) that breaks down precisely for heterogeneous sources.

  • Once the policy is treated as an unconstrained function over X×Y\mathcal{X} \times \mathcal{Y} and reparameterized via f(x,y)=log⁡π(y∣x)πref(y∣x)f(x, y) = \log \frac{\pi(y|x)}{\pi_{\text{ref}}(y|x)}, the stage objective (Eq. 4) is independent of πref\pi_{\text{ref}}, making Propositions 3.1–3.2 an immediate algebraic consequence of multiplying Boltzmann distributions.
  • More critically, Assumption 3.1 assumes fi⋆(x,y)f_i^\star(x, y) is uniquely identified (up to c(x)c(x)) for all (x,y)∈X×Y(x, y) \in \mathcal{X} \times \mathcal{Y}. Yet the paper's core motivation is valuing heterogeneous sources (e.g., sharegpt, truthful_qa, flan_v2_cot) whose prompt supports Pi(x)P_i(x) are largely disjoint or non-identical. For any out-of-domain prompt x∉supp(Pi)x \notin \text{supp}(P_i), the population objective of source ii imposes zero constraint on fi⋆(x,⋅)f_i^\star(x, \cdot), so Assumption 3.1 cannot hold. In practice, coalition behavior is governed entirely by parametric generalization and cross-stage interference/catastrophic forgetting—phenomena absent from the theory. W2. Mismatch between sequential alignment and standard pooled alignment.
  • Practitioners align LLMs by pooling preference sources into a single DAA run with fixed reference π0\pi_0, not by chaining mm sequential DAA runs with shifting reference policies.
  • Even under exact population optimization, the sequential sum ∑i∈Sfi⋆\sum_{i \in S} f_i^\star does not equal the minimizer of the pooled objective ∑i∈SwiLi(π;π0)\sum_{i \in S} w_i \mathcal{L}_i(\pi; \pi_0). Thus, the paper values contributions to a non-standard sequential training game rather than standard alignment. Only a minimal sanity check on a 135M model (SmolLM-135M) is provided in Appendix D.2, where ρseq/pool\rho_{\text{seq/pool}} already grows with coalition size (1.20→1.431.20 \to 1.43). W3. No ground-truth validation of the resulting Shapley values and no baselines.
  • Section 5 computes reconstructed Shapley values for n=9n=9 sources (Figure 2), but provides zero quantitative validation that these Shapley values are accurate or useful:
    1. There is no comparison against true Shapley values (e.g., exact coalitions on a smaller subset n=5n=5, or Monte Carlo permutation estimates) computed from actually trained coalition policies—either sequential or pooled—using their own generated responses.
    2. There are no downstream data-valuation benchmarks (e.g., source removal/addition curves showing that dropping high- vs. low-Shapley sources impacts alignment reward as predicted).
    3. There are no comparisons against natural baselines such as Task Arithmetic / LoRA weight averaging [18, 42], Leave-One-Out (LOO) valuation, or gradient/influence-based methods [24, 30, 39]. W4. Restrictive evaluation proxy (64-token Best-of-16 reranking) and potentially misleading error normalization.
  • Because the method cannot sample autoregressively from reconstructed policies (Appendix F.2), coalition utility in Section 5 is evaluated solely by scoring K=16K=16 candidate responses sampled from the unaligned base policy π0\pi_0, capped at 64 new tokens (Appendix C.6). Capping responses at 64 tokens truncates most realistic instruction-following answers, and restricting candidates to π0\pi_0 ignores distribution shifts induced by alignment.
  • In Figure 1, discrepancies are normalized by the retraining floor dfloord_{\text{floor}}. For Qwen2.5, Llama-3.2, and Phi-3.5, dfloord_{\text{floor}} reaches 1515–5050 in unnormalized log-probability units (compared to ∼0.7\sim 0.7 for SmolLM), likely because unnormalized sequence log-probabilities scale with length and stage-wise early stopping introduces high seed variance. A large denominator makes ρ≤2\rho \le 2 easy to satisfy without guaranteeing accurate downstream Shapley rankings.
Questions for authors
  1. Ground-truth Shapley fidelity: On a smaller player set (e.g., n=5n=5 or 66 where 2n≤642^n \le 64 runs are feasible), what is the Spearman/Pearson correlation between your reconstructed Shapley values and the exact Shapley values obtained from (a) standard pooled DAA policies generating their own responses, and (b) sequential DAA policies generating their own responses?
  2. Comparison with parameter-space merging: How does your log-probability reconstruction compare against simply averaging or adding task vectors θS=θ0+∑i∈Sλ(θi−θ0)\theta_S = \theta_0 + \sum_{i \in S} \lambda (\theta_i - \theta_0) [18]—which does yield a generative policy at the exact same cost of nn singleton runs?
  3. Downstream utility validation: If you rank the 9 UltraFeedback sources by their reconstructed Shapley values and cumulatively add/remove top vs. bottom sources in standard pooled IPO/DPO training, do the resulting generative models follow the predicted performance trajectory?
  4. Why is dfloord_{\text{floor}} an order of magnitude larger on Qwen2.5/Llama3.2/Phi-3.5 than on SmolLM (Figure 1)? Does ρrecon≤2\rho_{\text{recon}} \le 2 still hold if preference gaps are length-normalized or if downstream reward-model agreement (rather than raw log-prob gap) is measured?
  5. Effect of 64-token cap: Why were candidate generations restricted to 64 tokens in Section 5, and do the source valuations change under realistic generation lengths (e.g., 512–1024 tokens)?
6 comments

We thank the reviewer for their careful reading and constructive questions. We address each point below in the order raised. Throughout our answers, all references to sections, tables, or appendices refer to the paper revision submitted with this rebuttal, which incorporates all relevant suggested updates. A summary of these changes is provided in the general comment titled Paper revision.

Weaknesses [1/4]

W1. We agree that Assumption 3.1 does not hold in practice, and we do not claim that it does. Its role is rather to define an idealized regime in which sequential alignment on a coalition provably reduces to singleton alignments. This regime explains why coalition scores should be reconstructible from singleton policies. Whether this reconstruction remains accurate under finite neural training is therefore an empirical question, to which Section 4 is devoted. We answer the three points of W1 in turn.

  • (a) The reviewer notes that Propositions 3.1 and 3.2 follow from simple algebra. We agree, and this is by design. Our contribution is precisely to have found an alignment procedure under which coalition scores follow from singleton scores by simple algebra. This is not automatic, since the algebra closes only because each stage uses the policy produced by the previous stage as its reference. It is this procedure that reduces 2n−12^n-1 alignments to nn, which the reviewer lists as Strength 2.

  • (b) The reviewer argues that Assumption 3.1 cannot hold for heterogeneous sources. We agree that its uniqueness part fails in this setting. Indeed, for strictly convex losses such as IPO and DPO, uniqueness holds if and only if, at every prompt, the response pairs compared by the source form a connected graph on Y\mathcal Y. Heterogeneous sources violate this condition, since the objective of source ii does not constrain fi⋆(x,⋅)f_i^\star(x,\cdot) at prompts outside its support, nor at responses it never compares. We assumed uniqueness so that each coalition has a single exact policy, whichever minimizer each stage returns. The coalition utility is then unambiguously defined, and Proposition 3.1 can state that every exact sequential run returns this policy. However, this is the only role of uniqueness, and the reconstruction itself holds without it. Appendix F.3 of the revised paper establishes both points. Lemma F.1 proves the uniqueness characterization stated above. Proposition F.1 then shows that, as long as each singleton objective attains its minimum, the reconstructed coalition policy remains an exact output of the sequential procedure for every ordering, and still satisfies the reconstruction identity of Proposition 3.2. Without uniqueness, the reconstructed policy is the exact output in which every stage selects the same minimizer as the corresponding singleton. This is the natural choice, since each stage is initialized at its reference policy, as described in Appendix C.4, and therefore solves, in score coordinates, the singleton problem from the same starting point. The reconstruction thus does not require fi⋆(x,⋅)f_i^\star(x,\cdot) to be identified outside the support of source ii. We will therefore replace Assumption 3.1 in the camera-ready version with the weaker assumption that each singleton objective attains its minimum, and state the results of Section 3 for the coalition policy obtained when each stage applies the update of the corresponding singleton.

  • (c) The reviewer argues that coalition behavior is governed by parametric generalization and cross-stage interference, which the theory leaves out. We agree that the idealized model of Assumption 3.1 does not describe these effects. Rather, it predicts two properties, order invariance (Proposition 3.1) and reconstruction from singletons (Proposition 3.2), and each of these effects, like any other feature of finite training, could break either property. Section 4 tests both properties under finite neural training. It uses preference pairs whose prompts are never seen in training, so the measured behavior reflects generalization. These prompts are drawn from all sources and therefore include prompts outside each singleton's training source. For all base models, coalition sizes and source orders, both the order discrepancy and the reconstruction discrepancy stay below twice the variability between identical runs that differ only in their seed (Figure 1). Hence, within this tolerance, the idealized model accounts for the behavior measured under finite training.

Weaknesses [2/4]

W2. We agree that sequential and pooled training are different procedures, with different minimizers even at the exact optimum. The paper states this in Appendix D.2, and our values refer to the sequential procedure. We argue below that this procedure is a legitimate target for valuation, and we clarify the purpose of the comparison in Appendix D.2. We answer the two points of W2 in turn.

  • (a) The reviewer argues that practitioners pool sources under a fixed reference, that the sequential policy differs from the pooled minimizer even at the exact optimum, and hence that the paper values a non-standard training game. We agree with the first two statements, but not with the conclusion. Data values are always defined relative to a learning procedure [1], so the choice of procedure is a modeling decision rather than a mismatch. We choose sequential alignment because it is the procedure under which coalition scores follow from singleton policies. No such structure is known for pooled training, where each coalition requires its own alignment run. Moreover, this procedure is not specific to our work. As already discussed in our extended related work in Appendix A.5, trust-region methods such as TR-DPO [2] update the reference to the current policy during training, and report gains over fixed-reference DPO, IPO, and KTO. Our procedure corresponds to their hard-update variant when the data are ordered by source, and the reference is updated at source boundaries. Similarly, curriculum DPO [3] trains on successive blocks of preference pairs with the previous model as reference, and iterative DPO methods [4, 5] initialize and reference each round on the previous model. Continual alignment methods [6] also align on preference data that arrive over time, which is the setting of the data markets that motivate source valuation.
  • (b) The reviewer notes that the check of Appendix D.2 is limited to SmolLM-135M, and that ρseq/pool\rho_{\text{seq/pool}} grows with coalition size. This experiment does not aim to show that the two procedures are equivalent, which our theory does not imply. Its purpose is to check that replacing pooled training with the sequential procedure does not substantially change alignment behavior, so that values computed for the sequential procedure remain informative about pooled practice. Since this check complements our main claims, we ran it on the smallest base model of Section 4. On SmolLM, the discrepancy between the two procedures remains close to the variability between identical runs that differ only in their seed, and the two procedures prefer different responses on only a small fraction of held-out preference pairs. This discrepancy grows moderately with coalition size, but the induced preference decisions remain almost unchanged. Following the reviewer’s suggestion, we ran exactly the same protocol on Qwen2.5-0.5B-Instruct and added the results to Table 1 in Appendix D.2. They follow the same trend as for SmolLM.

Weaknesses [3/4]

W3. We thank the reviewer for these relevant suggestions, which strengthen the empirical support of the paper. Accordingly, Appendix D.3 of the revised paper reports a new experiment with n=5n=5 sources, in which every coalition is explicitly trained using the sequential procedure. The Shapley values computed from these trained coalitions are what our reconstruction aims to recover, and we call them explicit Shapley values. The experiment compares the reconstructed values with the explicit values, evaluates them on addition and removal benchmarks, and compares our reconstruction with Task Arithmetic. We answer the three points of W3 in turn.

  • (a) The reviewer asks for a comparison with Shapley values computed from explicitly trained coalitions. For each base model, we explicitly train all 25−1=312^5-1=31 coalitions and compute their Shapley values exactly. We train the coalitions with the sequential procedure, since the Shapley value that we estimate is defined with respect to this procedure, as discussed in our answer to W2. We also evaluate them with the utility of Section 5, so that the comparison measures the error of the reconstruction alone. Evaluating each coalition on its own generations would instead change the utility, and hence the Shapley values themselves. Moreover, the explicit and reconstructed games share the base and singleton policies, so any discrepancy comes from coalitions of at least two sources. Across base models and reward models, the reconstructed values deviate from the explicit values by 0.5×10−20.5\times 10^{-2} to 0.9×10−20.9 \times 10^{-2} on average, in units of reward standard deviation as in Section 5. In comparison, the explicit values of the five sources range from 0.030.03 to 0.130.13. Table 2 reports these errors. Figure 4 displays the explicit and reconstructed values for Qwen2.5, and Figures 9, 11, and 13 display them for the other base models. The reconstruction also recovers the sign of every value and correctly orders every pair of sources that the explicit game separates with 95% confidence.
  • (b) Since all 31 coalitions are explicitly trained, the same experiment provides the requested downstream benchmark without additional training. We rank the n=5n=5 sources by their reconstructed Shapley values. We then measure the utility of the explicitly trained coalitions obtained by adding sources in this order, starting either from the source ranked highest or from the source ranked lowest. We do the same for removal from the full set of sources. In all 1212 combinations of base model and reward model, adding the two sources ranked highest increases the utility more than adding the two sources ranked lowest. Symmetrically, removing the two highest-ranked sources decreases it more. Figures 5, 10, 12 and 14 show the complete curves, and Table 4 reports the differences with their 95% confidence intervals.
  • (c) Our goal is not to propose a valuation rule that would compete with existing ones. Instead, we compute the Shapley value, and more generally any semivalue, of heterogeneous preference sources at the cost of nn singleton alignments rather than 2n−12^n-1 coalition alignments. The relevant baselines are therefore other methods that provide coalition utilities without training each coalition. Task arithmetic [7] is one such method, at the same cost of nn singleton alignments, and we add it to the experiment in point (a). We set λ=1\lambda=1, so that task arithmetic adds the singleton updates without rescaling. This choice mirrors, in parameter space, the addition our reconstruction performs on sequence log probabilities, and it involves no tuned parameter. Evaluated with the same utility, task arithmetic deviates more from the explicit values than our reconstruction in 1111 of the 1212 combinations. Table 3 also shows that this gap widens as coalition size increases. Two remarks qualify this comparison. First, task arithmetic yields a generative policy, whereas our reconstruction only scores fixed responses, as explained in Appendix F.2. Second, a tuned λ\lambda could reduce task arithmetic error. A fair comparison would then require selecting λ\lambda on coalitions and prompts disjoint from those used for evaluation, which lies beyond the scope of this paper. The other baselines the reviewer suggested target different quantities. LOO measures the marginal contribution of a source to the full set only. Gradient and influence methods [8, 9] estimate, from a single training run, the effect of individual examples on the trained model, as discussed in Appendix A.4. Like LOO, they measure a marginal effect with respect to the full training set rather than an average over coalitions. Comparing these quantities with explicit Shapley values would therefore measure how much the definitions differ, not how accurately the Shapley value is estimated.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Paper revision

We thank all reviewers for their valuable feedback. Together with the individual rebuttals, we have submitted a revised version of the paper in which all changes are highlighted in blue. Below, we summarize the main modifications, with the specific weaknesses and questions they address.

  • (Reviewers aRCk [W3, Q1, Q2, Q3] and ez31 [W1, Q2]). We have added a new experiment that compares our reconstructed Shapley values with explicit Shapley values. For each of the four base models, we explicitly trained all 25−1=312^5-1=31 coalitions of five sources with the sequential procedure, and computed their Shapley values exactly under the three reward models. We report value errors, sign and rank agreement with 95% confidence intervals over the evaluation prompts, and selection agreement and utility errors for every coalition size. Since all coalitions are trained, we also report addition and removal benchmarks, in which the sources are ranked by their reconstructed values and evaluated on held-out prompts. Finally, we compare our reconstruction with the Task Arithmetic baseline, which also requires the same nn singleton alignments. Experiment details are provided in Appendix D.3. The corresponding results are reported in Tables 2–4, with Figures 4–5 for Qwen2.5 and Figures 9–14 in Appendix E for the other base models. These new experiments are mentioned in the main paper (Section 5).

  • (Reviewer ez31 [W1, Q1]). We have evaluated the coalitions of Section 4 on exactly the 16-candidate sets of Section 5. For every base model, coalition size, and source order, we measure selection agreement and utility error between the reconstruction and the explicitly trained coalition, and compare them with the same quantities across runs that differ only in seed. Experiment details are provided in Appendix D.6, and the corresponding results are reported in Table 7. This new experiment is mentioned in the main paper (Section 5).

  • (Reviewer aRCk [W4, Q4]). We have added an analysis of the retraining floor used to normalize the discrepancies in Figure 1. We compute the ratio ρ\rho for two incorrect coalition policy estimates built from the same policies as the reconstruction. We also measure, for every base model, the response lengths and the size of the coalition updates, and recompute the reconstruction ratio after normalizing each pair by its length. We additionally report how often the reconstruction and the explicitly trained coalition prefer different responses. Details are provided in Appendix D.4, and the corresponding results are reported in Table 5. This analysis is mentioned in the main paper (Section 4.5).

  • (Reviewer aRCk [W4, Q5]). We recomputed the Shapley values for the nine sources for Qwen2.5 using two new candidate pools. The first uses candidates of up to 512 tokens instead of 64. The second draws candidates from the base and singleton policies, including responses from aligned policies. We provide experiment details in Appendix D.5, and report the corresponding results in Table 6. These new experiments are mentioned in the main paper (Section 5), and the choice of the 64-token cap is now explained in Appendix C.6.

  • (Reviewer aRCk [W2]). We extended the comparison between sequential and pooled training to Qwen2.5-0.5B-Instruct, using the same protocol as for SmolLM-135M-Instruct. We have also clarified the purpose of this comparison. We added the corresponding results to Table 1 in Appendix D.2.

  • (Reviewers aRCk [W1], ez31 [W2, Q3] and Qyww [W1, Q1, Q2]). We have added a theoretical analysis of the uniqueness part of Assumption 3.1. Lemma F.1 characterizes exactly when uniqueness holds, and shows that it fails for sources with different prompt supports. Proposition F.1 then proves that the reconstruction remains exact without uniqueness, as long as each singleton objective attains its minimum. The analysis is provided in Appendix F.3 and is mentioned in the main paper after Assumption 3.1 (Section 3.2). Since uniqueness is not needed, we will replace Assumption 3.1 in the camera-ready version with the weaker assumption that each singleton objective attains its minimum. Lemma 3.1 and Propositions 3.1 and 3.2 will then be stated for the coalition policy obtained when each stage applies the update that its source produces alone, as in Proposition F.1. Uniqueness will appear as a remark, which guarantees that every exact run returns this policy.

We thank the reviewers again for their constructive feedback, which we believe substantially strengthened both the empirical support and the theoretical presentation of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.