- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper asks whether a recipient center N, before any joint training and without seeing raw data, can predict whether adding a donor center M's data will improve or degrade its model on its own population. The authors define the recipient-specific pooling effect Δ_true. They estimate it with a two-message protocol: N sends its fitted logistic model, and M returns its summed gradient g_M at that model. N then combines g_M with its own diagonal Fisher and a Fisher-normalized validation gradient. The result is a score with a first-order term and a nonnegative curvature term, obtained from a one-step influence displacement and a second-order expansion of recipient risk. A recipient-only bootstrap (donor message held fixed) gives a scale, and the rule pools, declines, or abstains according. Theoretical and experimental claims supports the overall narrative of the paper.
Critical strengths
-
Correct target quantity. The paper isolates the right object: a signed, recipient-specific, counterfactual risk change rather than a distributional discrepancy. The argument for why discrepancy measures answer a different question is convincing and is borne out empirically. This is a genuine clarification relative to the Data Addition Dilemma line of work, which ranks sources by discrepancy.
-
Compelling curvature result. The explanation (Prop. 1, App. A.13) is simple and correct. The ablations are convincing: the first-order term stays at 0.45–0.63 accuracy as n₁ grows on Toy and degrades to 0.458 on Toy-Op, while the full score reaches 0.94–1.0. This sharpens what Basu et al. (2020) and Koh et al. (2019) leave implicit: second-order group influence on parameters is not the same as second-order change in target risk.
-
Lightweight, practical protocol. The protocol needs one O(d) message each way and O(d) scoring after an O(n₁d) Fisher diagonal. Because the full-Hessian variant is shown to add little, the diagonal approximation is empirically justified rather than merely asserted.
-
Honest scoping of the theory. The assumption accounting (App. A.1, A.11), the explicit statement that A4 is assumed rather than proved, the admission that Thm. 1 certifies harm but never benefit, and the separation of the fixed-donor and local-shift bootstrap analyses are unusually careful. The privacy appendix is also careful about adjacency with no overclaims.
-
Decision-theoretic evaluation. Danger versus over-caution errors, cost curves against trivial rules, and break-even abstention costs are more meaningful than AUROC alone for this application.
Critical weaknesses
1. Missing pooling baselines. Matching-based pooling (https://arxiv.org/pdf/2602.07154) admits donor data relative to a target centroid, at the domain level in its theory and at the sample level in its implementation, rather than discarding a donor entirely. The proposed rule is all-or-nothing: a ‘decline’ discards the entire donor even if most of it is compatible, which is wasteful when curation is expensive. A protocol-compatible variant seems feasible: the recipient sends a feature centroid along with θ_N, and the donor filters locally and returns the gradient over its retained subset, which is still two messages. How does the pooling effect of such filtered donors compare with the decline decisions?
2. The method can certify harm but not benefit, yet is framed as a pool/decline/abstain rule. The paper states this clearly in places: Thm. 1 recovers only positive signs, and Cor. 3(ii) shows pool is unreachable under the theoretical threshold. The population score is always ≥ 0, so negative scores arise only from the finite-sample first-order term, which the paper itself shows is weak. The benefit channel, term (B) in Lemma 10 (baseline excess risk ~d/n₁), is not represented in the score by any explicit variance-reduction term. The rule is therefore structurally biased toward declining, and the low danger rates are partly a consequence of this bias rather than evidence of discrimination. Two observations support this: κ ranges from 0.037 to 0.13 on Toy, so the score overestimates harm by roughly 8–25×, and DP noise inflates the score upward. The abstract and introduction should say plainly that the guarantees concern harm detection, and that "pool" decisions carry no guarantee in the fixed-donor regime.
3. Theory–experiment mismatch in the imaging settings. The guarantees assume unpenalized logistic fits with n₁ ≳ d (Lemma 1, Lemma 9). The imaging experiments use n₁ = 100 with frozen ResNet-50 embeddings, nominally d = 2048, so the unpenalized MLE generically does not exist. Appendix B.2 mentions "the same ridge term on its diagonal as ...," which suggests the deployed method is ridge-penalized, possibly after dimension reduction. Neither the ridge strength, its selection, nor any dimensionality reduction is reported in the main text. This matters in two ways: a) The theory does not cover the main experiments b) With a ridge term, the first-order optimality that makes the first-order term vanish at the baseline no longer holds in the same form. Please report d, the penalty, and its tuning, and state explicitly which results apply to the penalized estimator.
4. Abstention rates and break-even costs limit practical value. At k = 1, the rule abstains on 79–84% of trials on ISIC B, CheXpert, and Waterbirds, and the break-even abstention cost at r = 1 is 0.33–0.37. In the clinical scenario that motivates the paper, abstention means "collect more data" or "do the collaboration anyway." It is not obvious that abstaining in four of five cases is useful, especially when it must cost less than a third of a wrongly declined donor.
5. Evaluation labels and base rates. The help/hurt label is the sign of a loss difference on an evaluation fold of n_eval = 300 (150 for ISIC A). Many true effects are near zero, so the label is itself noisy. Base rates are highly skewed: 0.869 hurt on Toy, 0.802 on MNIST-Bar, 0.78 on ISIC A, 0.26 on Waterbirds. An effect-size-stratified analysis would clarify whether the method is detecting meaningful harm or mostly separating large-shift donors from clean ones. Examples include accuracy as a function of |Δ_true|, or AUROC excluding |Δ_true| below a tolerance.
6. The fixed pooling weight limits relevance. The decision is evaluated only at α = m/(n₁+m). In practice, and in the transfer-learning literature, one would reweight or shrink toward the recipient, under which "decline" is rarely optimal. The limitations section mentions this, but it weakens the practical motivation: a donor that harms at α = 0.8 may help substantially at α = 0.2.
Questions for authors
Please see "Critical Weaknesses" section of the review above. The paper is interesting and approaches the problem in a different way. I am keen to see the authors' response and am eager to revise my rating if the response addresses all my concerns.