- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies source-level Shapley valuation for preference-based LLM alignment. It defines coalition models through sequential preference optimization, using each stage's policy as the next stage's initialization and reference. Under an attained and identifiable population optimum over the score space, each source induces a reference-independent relative log-probability increment. Coalition complete-response log-probabilities can then be reconstructed, up to a prompt-dependent constant, by adding singleton log-policy ratios to the base model's scores. For shift-invariant evaluation protocols, this reduces the number of trained models from one per coalition to one per source; exact Shapley summation still requires all coalition utilities. Experiments examine order sensitivity and reconstruction discrepancies for nine UltraFeedback provenance sources, using IPO on four base models and supplementary DPO experiments on two. The application computes exact Shapley sums for a reconstructed game that selects among 16 fixed base-model candidates and evaluates selections with three reward models.
Critical strengths
S1. A precise way to remove coalition-specific alignment for a defined valuation game. Lemma 3.1 and Propositions 3.1-3.2 show how reference-independent score increments produce an order-invariant coalition policy and how singleton policies encode its complete-response scores. Section 3.4 identifies the class of shift-invariant utilities for which the unknown normalization is irrelevant. Under these assumptions, the main benefit is avoiding coalition-specific retraining while preserving the specified utility. Exact Shapley computation still requires exhaustive utility enumeration.
S2. The empirical study tests the structural predictions and includes useful controls. Figure 1 compares both order changes and reconstruction against repeated-training variability across four models. Appendix D.1 repeats the tests with DPO, and Appendix D.2 distinguishes the sequential procedure from pooled IPO, reporting 1.3%-3.0% disagreement on held-out preference-pair rankings for SmolLM (Table 1). These checks provide preliminary evidence beyond the idealized derivation and help delineate what the proposed alignment procedure measures.
Critical weaknesses
-
W1. The reported validation does not establish fidelity for the actual utilities and source values. My main concern is a mismatch between the validation metric and the valuation application. Sections 4.2-4.5 measure mean absolute errors of the original held-out preferred/rejected log-probability gaps and report discrepancies within twice a three-seed retraining reference. Section 5 and Appendix C.6 apply an argmax to 16 newly generated candidates and evaluate the selected response's reward. Agreement on the original pair does not constrain the other candidates' rankings, and argmax utilities can change sharply near a decision boundary. Neither Figure 1 nor the DPO counterpart reports selection agreement or reward-utility error on these candidate pools. The tests also cover only three nested coalitions, while Figure 2 aggregates marginal utilities over all 512 coalitions without reported valuation uncertainty. The application therefore lacks direct validation of finite-training source-value fidelity. Evaluating the existing coalition models on the same candidate pools, together with an exact small-source Shapley comparison and uncertainty estimates, could substantially alleviate this concern.
-
W2. The practical scope of the identifiable population model needs clarification. Assumption 3.1 requires each source's minimizers to agree on every response's relative score for every prompt. This is substantially stronger than exact optimization. For example, with one prompt, three responses, beta = 1, and only the comparison a preferred to b, the score vectors (1/4, -1/4, 0) and (1/4, -1/4, 1) both achieve zero IPO loss but induce different probabilities for the unobserved response. Exact optimization alone therefore does not identify a source-specific update. The highest/lowest completion construction in Appendix C.1 does not establish the necessary population comparison coverage; likewise, a fixed neural policy class does not automatically realize the exponential tilts used in Appendix B. The acknowledged idealization would be more informative with sufficient data-support conditions and an explanation of how unconstrained directions are selected in practice. This concern limits the theorem's explanatory scope; it does not contradict the propositions under their assumptions.
Questions for authors
- Q1. Validation of candidate selection (W1). For the already trained coalitions and orders, what are the response-selection agreement and standardized reward-utility discrepancies between reconstruction and explicit sequential models on exactly the 16-candidate pools used in Section 5? Please compare these discrepancies with same-procedure seed variability.
- Q2. Validation of source values (W1). In a three-source subgame with all eight coalition utilities evaluated explicitly under a specified ordering/seed protocol, how close are the reconstructed and explicit Shapley values? Please report value errors and sign/rank stability, with uncertainty from training and evaluation sampling.
- Q3. Clarification of the assumptions (W2). What sufficient conditions on each source's prompt support and response-comparison distribution ensure Assumption 3.1, including finite attainment for DPO? Does the implementation impose any reference-independent rule on score directions left unconstrained by observed comparisons, and what policy-class realizability is needed for the exponential-tilting argument?
5 comments
Weaknesses [2/2]
W2. The reviewer makes three points about Assumption 3.1. First, exact optimization does not imply uniqueness, as the example with three responses shows. Second, comparing only the highest and lowest completions of each prompt does not provide the coverage that uniqueness requires. Third, a neural policy class need not realize the updates used in the proofs. The reviewer therefore asks how the directions left unconstrained by the data are selected in practice. We thank the reviewer for these observations, which are correct and helped us clarify the theoretical role of Assumption 3.1. We do not claim that this assumption holds in practice. Its role is to define an idealized regime in which sequential alignment on a coalition provably reduces to singleton alignments. This regime explains why coalition scores should be reconstructible from singleton policies, and it predicts the properties that we then test under finite neural training in Section 4. We now state this role explicitly after Assumption 3.1, and we address the three points of the reviewer in the new Appendix F.3.
-
(a) When uniqueness holds. We prove in Lemma F.1 the exact condition for uniqueness. For IPO and DPO, the minimizers of a source are unique up to a prompt-dependent constant if and only if, at every prompt, the response pairs compared by the source form a connected graph. In the reviewer’s example, the third response is isolated, so uniqueness fails. For the same reason, it fails for our sources, since each source compares a single pair per prompt and the sources have different prompt supports.
-
(b) Why we assumed uniqueness, and why the reconstruction does not need it. We assumed uniqueness so that each coalition has a single exact policy, whichever minimizer each stage returns. The coalition utility is then unambiguously defined, and Proposition 3.1 can state that every exact sequential run returns this policy. However, this is the only role of uniqueness. We prove in Proposition F.1 that the reconstruction remains exact without it, as long as each singleton objective attains its minimum. We fix one minimizer for each source, and consider the run of the sequential procedure in which every stage selects the minimizer of the corresponding singleton. We show that this run is exact for every ordering, that all orderings return the same policy, and that this policy satisfies the reconstruction identity of Proposition 3.2. In the camera-ready version, we will therefore replace Assumption 3.1 by this weaker assumption, and state the results of Section 3 for the coalition policy obtained when each stage applies the update of the corresponding singleton, with uniqueness as a remark.
-
(c) What happens in practice. Our training adds no constraint on the scores that the data leave free, such as those of responses that a source never compares. However, we initialize each stage at its reference policy, as described in Appendix C.4. Each stage therefore starts from a zero update relative to its reference, exactly as the singleton alignment starts from the base policy. This mirrors the selection described in Proposition F.1, although finite training only approximates it. As for realizability, we state the theory over all full-support policies, and we do not assume that a finite neural model realizes each update. Whether finite training behaves like the idealized regime is therefore an empirical question. Section 4 shows that, under finite training, coalition scores remain as close to their reconstruction as two runs that differ only in their seed, and Appendices D.1 and D.6 extend this test to DPO and to the quantities of Section 5.
Questions
Q1. The reviewer asks for the selection agreement and reward-utility discrepancies between the reconstruction and the explicit sequential models on the 16-candidate sets in Section 5, compared with seed variability. This is a relevant test, since it evaluates the reconstruction directly on the objects of Section 5. We measured these quantities and report them in Appendix D.6. We used the coalitions, orders, and seeds from Section 4, along with the candidate sets, reward models, and standardization from Section 5, so no additional training was needed. We observe that the reconstruction selects the same candidate as the explicit coalition on 68.6% to 87.1% of the prompts, compared with 72.3% to 88.7% for two runs that differ only in their seed. The ratio of reward discrepancies, which we build as in Section 4, lies between 1.08 and 1.62. The utility errors, between 1.0 and 3.2 in reward standard deviations, are of the same order as those between seeds, between 0.5 and 3.0 . Both are of the order of the uncertainty due to the finite number of evaluation prompts. Table 7 reports all values.
Q2. The reviewer asks for a subgame in which all coalitions are explicitly trained, with value errors and sign and rank stability. We found this suggestion particularly valuable, and we followed it on five sources rather than the suggested three. We explicitly trained the 31 resulting coalitions for the four base models, computed their Shapley values under the three reward models, and report the comparison in Appendix D.3. We observe that the reconstructed values deviate from the explicit values by 0.5 to 0.9 on average, whereas the explicit values range from 0.03 to 0.13. The reconstruction recovers every sign and correctly orders every pair of sources that the explicit game separates with 95% confidence. Its Spearman correlation with the explicit values lies between 0.9 and 1.0, and between 0.6 and 0.8 for Phi-3.5, whose five values are close to each other. Table 2 reports these results with 95% confidence intervals over the evaluation prompts. We trained each coalition of Appendix D.3 with a single seed, so these intervals only account for the finite number of evaluation prompts. We measured training variability at the level of coalition utilities in Appendix D.6, where the errors of the reconstruction are of the same order as those between seeds, as detailed in our answer to Q1.
Q3. The reviewer asks for sufficient conditions for uniqueness and attainment, for the realizability required by the proofs, and for the rule applied to unconstrained directions. These questions helped us make the theory's scope explicit. For uniqueness, we prove in Lemma F.1 a necessary and sufficient condition, namely a connected comparison graph at every prompt. For attainment, IPO needs no condition, since its quadratic loss always has a minimizer, as we now state in Appendix F.3. For DPO, the logistic loss attains its minimum if and only if, within every connected component of the comparison graph of a prompt, no group of responses wins every comparison against the others. This is the classical condition for the existence of the Bradley–Terry estimator. It fails for our sources, since each prompt has a single preferred response. Our DPO results in Appendix D.1 therefore provide evidence under finite training rather than a test of the exact regime. Finally, we discuss realizability and the rule on unconstrained directions in our answer to W2-(c).
If our responses and the revised paper address the reviewer's concerns, we would be grateful if they would consider updating their assessment. We would be happy to clarify any remaining points during the discussion period.
We thank the reviewer for their careful reading and constructive questions. We address each point below in the order raised. Throughout our answers, all references to sections, tables, or appendices refer to the paper revision submitted with this rebuttal. A summary of these changes is provided in the general comment titled Paper revision.
Weaknesses [1/2]
W1. The reviewer argues that Section 4 validates the reconstruction on log-probability gaps between two fixed responses, whereas Section 5 relies on a different object, namely the candidate selected among 16 and its reward. Agreement on the original pair does not constrain how the other candidates are ranked, and an argmax can change abruptly near a decision boundary. The reviewer also notes that Section 4 covers only three nested coalitions, and that Figure 2 reports no uncertainty. We thank the reviewer for this precise analysis, and we agree that the validation should cover the quantities used in Section 5. The three checks suggested by the reviewer are well suited to this question. We carried out all of them, and they substantially strengthen the paper's empirical support.
(a) Validation on the selections and utilities of Section 5. We evaluated the coalitions of Section 4 on exactly the 16-candidate sets of Section 5. We compared the reconstruction with the explicitly trained coalitions and used two runs that differ only in their seed as a reference. We observe that the reconstruction is as close to the explicit coalitions as a retraining is, both in the selected candidates and in the utilities. The sensitivity of the argmax near decision boundaries therefore affects the reconstruction no more than an independent retraining. We report this experiment in Appendix D.6, and our answer to Q1 gives the numbers.
(b) Exact Shapley values on a small set of sources. We explicitly trained all 31 coalitions of five sources. We observe that the reconstructed values recover the signs and the order of the sources, with small errors. We report this experiment in Appendix D.3, and our answer to Q2 gives the numbers.
(c) Uncertainty. We report 95% confidence intervals over the evaluation prompts in Appendices D.3 and D.6, and we compare every discrepancy in Appendix D.6 with the variability between runs that differ only in their seed. We also computed 95% confidence intervals for the nine values of Figure 2 in Appendix D.5. We observe that every value that differs significantly from zero keeps its sign when the candidate sets change.