- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies retraining and reselection after data are removed from the training set. In particular, such a process may include different levels of choices: 1. parameter changes, such as different gradients in training. 2. configuration changes, such as due to different validation results. The paper's key observation is that you may early stop if you observe that the final range of candidates is narrow and drop the ones that cannot win. The range is predicted through gradient descent, and the curvature bound is assumed, assuming the objective is convex.
Critical strengths
The setting is interesting, and the observation could be useful for saving training time.
Critical weaknesses
The paper is very hard to follow without an LLM's help. The goal of each section is very hard to follow. The authors should add an overview of the goal for each section. The terms should be defined or explained, like "menu", "race," and "rectangular". There is no explanation of what theorems 2, 4, and 5 are trying to achieve. The theorems are very hard to parse. The experiment section is also very unstructured and hard to read. The answers to the proposed three questions at the beginning are very hard to find.
Questions for authors
See weakness.
2 comments
Revision: definitions, theorem explanations, and experimental structure
Thank you for identifying the exposition problems. The revision, marked in blue, adds definitions, a section roadmap, theorem explanations, three main-text figures, and experimental headings that answer the motivating questions.
- State the problem and define the terminology. The training procedure fits several configurations, selects one using validation data, and reports its prediction. Deleting observations can change both the fitted parameters and the configuration selected. The attribution is the prediction change after repeating this complete procedure. Exactly refitting only the old winner can therefore answer a different question.
Section 1 now defines a “menu” as a fixed, ordered list of configurations, such as regularization strengths. Section 3 defines “rectangular” information as separate intervals for validation scores and outputs, without their joint dependence through the fitted parameter. Section 4 defines a “race” as repeated refinement and certified exclusion of candidates that cannot win.
Section 4 also clarifies that the algorithm does not predict future gradients. Newton supplies successive parameter iterates. The certificate uses the actual retained-data gradient and a known strong-convexity bound. The resulting intervals describe optimization uncertainty, not statistical confidence about a population.
- Explain what Theorems 2, 4, and 5 are trying to achieve. Theorem 2 answers: which selected attributions are still possible? It combines the outputs of candidates that can win, respecting the specified tie order. Under the stated independent interval information, the best worst-case absolute-error radius is half the width of the resulting hull.
Figure 1 and the two-candidate example make this concrete. Suppose validation intervals overlap, while output intervals are and . With a known zero baseline, the selected attribution lies in . Its midpoint has error at most 0.015, so tolerance 0.02 is met without knowing which model wins. Changing the second output interval to prevents that conclusion. Output agreement can therefore determine termination before score separation.
Theorem 4 answers: does refinement preserve correctness? With valid input bounds, the true selected candidate is never removed, and every returned interval contains the target. Every returned point estimate meets the requested tolerance. A bound conditional on winning need only retain the actual winner's output. Unresolved cases retain an enclosure without an unjustified point estimate.
Theorem 5 answers: how much refinement is sufficient? Under its stated synchronous schedule and shrinking interval widths, the bound depends on prediction disagreement among near-optimal candidates. If their predictions already agree sufficiently, separating their validation scores is unnecessary. The theorem explicitly separates this work guarantee from the safety statement.
- Organize the experiments around the questions. Section 5 now uses the following result headings.
“Does reselection change attribution?” Winners change in 12.3–19.1% of queries by setting. On switched queries, exact refitting of the old winner gives 55.7–64.8% mean sign accuracy. This isolates a change in the attribution target from ordinary approximation error.
“When does output stopping save refinements?” Figure 3 and Table 2 show 1.49–12.36% fewer scalar refinements in the probability study, versus 0–2.60% for joint outputs. The 4,800 stringent joint-logit queries retain their zero additional savings. This comparison holds the certificate tolerance and candidate-pruning procedure fixed.
“Do fewer refinements reduce query time?” Rectangular stopping takes 95.1–98.1% of the winner race's pooled time for scalar queries. Conditioning is slower, so the revision identifies rectangular stopping as the practical default for these workloads. Figure 2 places the full-Newton control beside Proposition 6 and makes its solver dependence visible.
- Make the section purposes and contribution easy to locate. The introduction now explains the progression: Section 2 defines the counterfactual, Section 3 constructs its enclosure, Section 4 establishes correctness and refinement bounds, Section 5 answers the empirical questions, and Section 6 distinguishes the contribution from prior work. Table 3 identifies the established ingredients from Suzuki, Xiang, Wilson, and Jamieson–Talwalkar beside the present results.
These changes address the presentation issues while preserving the mathematics and data. We would appreciate reassessment of the revised version. If a specific assumption, equation, or proof motivates the significant soundness concern, please identify it so that we can respond precisely. The written weaknesses currently concern exposition rather than a specified mathematical defect.
Clarifying the problem, terminology, and theorem roles
Thank you for identifying the exposition problems. We explain the main idea and theorem roles below, then specify the planned changes.
The problem. The training procedure selects among fitted configurations using validation data. Deleting a training group can change both the fitted parameters and the selected configuration. We seek the change in the selected prediction after repeating that procedure. Refitting only the old winner can answer a different question. Proposition 1 and the fixed-exact baseline isolate this from approximation error.
Terminology and assumptions. A “menu” is a finite ordered list of configurations, such as 19 regularization strengths. A “race” refines candidate fits and eliminates candidates proved unable to win. It is not a stochastic competition. “Rectangular” means using separate scalar intervals for validation scores and outputs, without their joint dependence through the same parameter. Conditioning uses that dependence by requiring a winning candidate's validation loss to be below the best available upper bound.
Damped Newton supplies iterates, not predicted future trajectories. The certificate uses the actual retained-data gradient and a known strong-convexity lower bound. Strong convexity, not convexity alone, is required for the residual region. The intervals are deterministic optimization enclosures, not statistical confidence intervals. The float64 implementation is evaluated numerically, not formally verified.
For example, two candidates can have overlapping score intervals but output intervals [0.49,0.51] and [0.50,0.52]. With known zero baseline, the selected attribution is in [0.49,0.52] regardless of the winner. Tolerance 0.02 is satisfied. If the second output interval were [0.90,0.92], it would not be. Thus identity and output uncertainty differ.
Theorem 4 proves safety during refinement and elimination. With valid input bounds, the true winner remains and every returned interval contains the target. A bound conditional on winning need only retain the actual winner's output. An unresolved computation returns no point estimate. This proves safety, not finite termination in every tie case.
Theorem 5 bounds refinement work. Under the stated synchronous schedule and shrinking score/output intervals, the bound depends on prediction disagreement among near-optimal candidates. Agreement can make score separation unnecessary. Disagreement beyond the tolerance requires more information. The contraction assumption belongs to this work bound, not the safety theorem.
Does output stopping save refinements? Sometimes. It saves none on 4,800 stringent joint-logit queries. The probability study gives 1.49–12.36% scalar savings and 0–2.60% joint savings against an equally certified winner race. These depend on tolerance and target count.
Do those savings lower query time? Rectangular stopping gives modest scalar pooled-time savings of 1.9–4.9%. Conditioning is slower overall. Point estimates at matched work are already accurate. Certification justifies stopping rather than improving the observed point estimate.
Established regions and selection guarantees remain credited to Suzuki et al. (2014), Xiang et al. (2017), and Wilson et al. (2020). We will also distinguish predictive-model reselection from tuning an attribution estimator, as studied by Wang et al. [1].
Could you identify the assumption, equation, or proof underlying the significant soundness concern? The written weaknesses identify presentation problems, which we accept, but no specific mathematical defect. We would appreciate a specific objection, or reconsideration of the soundness assessment in light of these clarifications.
[1] Wang et al., “Taming Hyperparameter Sensitivity in Data Attribution,” NeurIPS 2025.