- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies credit assignment for AI-generated content that uses multiple creators’ or webpages’ content in context. It models the problem as a coalitional game with an unknown ground-truth value function and an estimated value function , and uses the least core as the allocation concept to minimize the largest coalition under-compensation. The paper derives a sensitivity bound based on balanced distributions over coalitions, proposes joint LP/QP constraint-generation algorithms with sampling-based and LLM-based separation oracles, and discusses minimal winning coalition seeding for binary value functions. The experiments use LLM-based binary question-answering evaluations on BrowseComp-Plus and a synthetic web-retrieval dataset, comparing random sampling, constraint generation, LLM seeding, LLM separation, and zero-shot allocations. The authors report that constraint generation can achieve comparable holdout deficit to the Yan--Procaccia sampling baseline with fewer value-function queries, while LLM-based variants are faster but may produce sparser and less stable allocations.
Critical strengths
- The problem formulation is timely and potentially important. Modeling context attribution as a cooperative game gives a clear incentive-oriented interpretation of credit assignment. The least core is a meaningful alternative to purely marginal or interpretability-oriented attribution because it explicitly considers whether coalitions are under-compensated.
- The algorithmic framework is modular and potentially useful. The constraint-generation formulation separates the optimization problem from the coalition-search oracle, allowing random, domain-specific, or LLM-based separation procedures. The paper also explores the tradeoff between faster LLM-guided search and broader egalitarian allocations across multiple experimental variants.
Critical weaknesses
- The central theoretical claims are not sound as written. Example 3.3 is inconsistent with its own definition: the displayed conjunction implies that only for , which gives , not . The claimed counterexample appears to require a disjunction instead, together with a different disagreement-set expression. More substantially, Theorem 3.1 appears to use a dual characterization that ignores the primal nonnegativity constraints on . Proposition B.1 lacks a monotonicity assumption on , so the proposed marginal allocation can be negative. Corollary C.1 also appears to promote a -probable sampled-feasibility guarantee to an exact global -optimality statement. These issues affect the theoretical foundation of the paper rather than being minor presentation errors.
- The empirical evidence does not establish the claimed least-core approximation or query-efficiency advantage. For , the evaluation uses only 212 uniformly sampled holdout coalitions and a 99th percentile, even though the least core is defined by the worst coalition over possibilities. The Yan--Procaccia baseline is itself a random-sampling approximation, so matching it does not demonstrate closeness to the true least core. The supplementary package contains 69 synthetic instances with 34--42 documents per row, while the paper reports and 32 instances per dataset, without a split or run manifest. The released prompts expose the Gold Answer to the LLM-based initializer, oracle, and zero-shot methods, which would create label leakage if those prompts were used literally. The call-count accounting, model randomness, and cross-model robustness are also not sufficiently specified.
Questions for authors
Clarifications that materially affect the assessment
- Is the conjunction in Example 3.3 a typographical error for a disjunction? Under the formula currently printed, only for , so . Please provide the corrected statement and proof.
- Can the authors provide a complete dual derivation for Theorem 3.1 that explicitly accounts for ? In particular, why should all coalition inclusion loads be equal rather than bounded by a common dual variable? Does the claimed bound still hold for arbitrary normalized value functions ?
- Were the Gold Answers actually passed to the LLM-based initializer, separation oracle, and zero-shot allocation prompts? If yes, why is this information available to the algorithm and how is the comparison with the random baseline fair? If no, please provide the exact prompts used in the experiments.
- What is the exact mapping from the 69 synthetic records in the supplementary package to the reported and experiments and the claimed 32 instances per dataset? Please provide the filtering, splitting, and run manifest.
- Please report whether each coalition evaluation is deterministic or cached, the model version, temperature, seed, retry behavior, and whether the full-document and empty-document coalitions are explicitly forced to satisfy and . Also report monotonicity violations of the independently evaluated binary .
- How are all LLM calls counted? Please clarify whether the reported counts include answer-generation calls, grading calls, initializer calls, LLM-oracle proposal calls, and zero-shot calls, and provide a complete quality-versus-cost table.
3 comments
Were the Gold Answers actually passed to the LLM-based initializer, separation oracle, and zero-shot allocation prompts? If yes, why is this information available to the algorithm and how is the comparison with the random baseline fair? If no, please provide the exact prompts used in the experiments.
Yes, those are the exact prompts, and this isn't label leakage. We assume the baseline answer is known and only want to assess the counterfactual of having a different subset of documents in context for that answer. In a real system, this answer might come from user feedback, such as a thumbs up or down. Every value function query, including YP's, grades against that same baseline answer. Also the LLM initializer and oracle only suggest subsets; we still score every suggestion with the value function and evaluate on our shared holdout set. Further, CG with random oracle + seeding never sees the answer in a proposal prompt, but still matches YP in far fewer calls.
What is the exact mapping from the 69 synthetic records in the supplementary package to the reported (n=10) and (n=32) experiments and the claimed 32 instances per dataset?
The supplement has all 69 synthetic questions. For both and , we took the first 32 questions in file order, kept their gold and evidence documents, and trimmed extra negatives to reach . We did the same for the first 32 rows of BrowseComp.
Please report whether each coalition evaluation is deterministic or cached, the model version, temperature, seed, retry behavior, and whether the full-document and empty-document coalitions are explicitly forced to satisfy (\hat v(N)=1) and (\hat v(\varnothing)=0). Also report monotonicity violations of the independently evaluated binary (\hat v).
Each subset is evaluated once per question and cached on disk, so every method sees the same (deterministic) value function. We fixed empty and full sets to 0 and 1 by construction. We used one LLM with default decoding, fixed seeds for training and holdout sampling, and no retries on errors (which score zero). Across over two million nested pairs on BrowseComp, roughly 99.5% satisfy monotonicity. Only 0.51% of pairs violate it overall, with a 0.24% median per question and zero violations from all-negative subsets. This supports our assumption in Section 2.
How are all LLM calls counted? Please clarify whether the reported counts include answer-generation calls, grading calls, initializer calls, LLM-oracle proposal calls, and zero-shot calls, and provide a complete quality-versus-cost table.
Each new subset evaluation takes two calls (answer and grading), each LLM oracle step takes one call, and ZS takes one call. We exclude the holdout set because no algorithm sees it during search. It's only a test set for scoring final allocations. Here's the (mean, non-egalitarian) n=32 results:
| Method | Seed | Oracle | Synthetic Deficit | Synthetic Calls | BrowseComp Deficit | BrowseComp Calls |
|---|---|---|---|---|---|---|
| YP | 0.50 | 8192 | 0.64 | 8192 | ||
| YP | S | 0.52 | 8256 | 0.71 | 8256 | |
| CG | R | R | 0.51 | 1796 | 0.65 | 1852 |
| CG | L | R | 0.54 | 195 | 0.66 | 1625 |
| CG | S | R | 0.64 | 256 | 0.72 | 412 |
| CG | R | L | 0.55 | 109 | 0.79 | 128 |
| CG | L | L | 0.56 | 137 | 0.80 | 211 |
| CG | S | L | 0.56 | 113 | 0.76 | 96 |
| ZS | 0.67 | 1 | 0.76 | 1 |
Apologies for the typo in my Review: "... only 2^{12} uniformly sampled holdout ...", not "212". But it's alright, we're done here. Thanks for your detailed response. It addresses my concerns. I will raise my score.
Thank you for your detailed review. We want to emphasize that we’ve fixed all mathematical issues with the main theoretical results. The main message of our paper and the takeaways from the theory have not changed.
This was a typo, as the conjunction indeed should be a disjunction. We’ve fixed this. We’ve also fixed the issue with the disagreement condition. It should read: the value functions disagree if and only if S=B∪T for T a proper subset of A. The fraction of (nonempty) coalitions on which the value functions disagree is exponentially small: it is (2n/2−1)/(2n−1)=1/(2n/2+1). And, the least core deficits (with the above fixes) are 1/2 and 0, respectively. So, the fixed example conveys the point we wanted to get across here.
Indeed the dual we wrote down was incorrect as your counterexample shows. We have included a fix: the statement of Theorem 3.1 holds under the assumption that the coalitional value functions are monotone (that is, additional creators do not hurt coalitional value). To argue this, we analyze the corrected dual with the correct inequality constraints corresponding to the primal nonnegativity constraints. A simple value-transfer argument shows that balanced distributions are dual optimal, which then allows us to arrive at the claimed sensitivity bound using the same argument as before.
Monotonicity is a reasonable assumption in our credit assignment setting. For example, in our web-retrieval setting, access to more documents can only improve the quality of a response. We’ve included the monotonicity assumption in Section 2.
We’ve added the monotonicity assumption to the problem formulation. It is needed for our fixed version of Theorem 3.1 and is a realistic assumption that holds in our AI credit assignment setting. Thank you for pointing this out.
We were a bit sloppy with the terminology here. Corollary 4.3 previously claimed exact ℓ2 optimality over all feasible schemes, whereas the proof only establishes optimality relative to the sampled constraints. We have restated it more precisely as an approximate optimality result.
First, our holdout set has 4,096 coalitions (212, not 212). Because checking every subset at n=32 is intractable, we also ran n=10 experiments which are presented in the appendix. You can see there that YP evaluates every subset (about ~2k calls) to find the exact least core, and on our synthetic dataset, CG with LLM seeding/oracle matches it on 31/32 questions in ~65 calls. At n=32, CG with a random oracle matches YP using four to five times fewer calls. The LLM oracle is faster still: on our synthetic dataset it got within about 0.06 of YP in just 110 to 140 calls (60 to 75x fewer), ahead of zero shot. On BrowseComp it did roughly the same as ZS, likely because only 64% of gold docs answer the question on their own. We used the 99th percentile at n=32 so one noisy LLM judgment, such as guessing from memory, doesn't push the deficit to 1.