- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes GhostMatch, a validation-free online batch selection method for LLM fine-tuning. At each step, a candidate batch ( b=8) is reduced to a weighted subset (k=4) by approximately reconstructing the candidate batch's mean gradient. Selection is formulated as a facility location problem over pairwise gradient distances, which yields a monotone submodular objective with the classical (1−1/e) greedy guarantee, while the underlying gradient geometry is computed exactly via ghost inner products. A nonconvex stationarity bound relates optimization error to the realized representation cost, with a linear-convergence variant under the PL condition. Experiments on four LoRA fine-tuning configurations (1.1B to 8B) claim parity with validation-guided baselines (GREATS/OPUS) while using no held-out data, plus an ablation showing that exact gradient geometry beats a zeroth-order approximation.
Critical strengths
-
A clean formulation with a well-controlled ablation. Casting per-step selection as facility location over exact, ghost-computed gradient geometry is an elegant combination, and the ZeroOrder ablation (16/16 matched runs) cleanly isolates the value of exact geometry, a design variable that prior validation-free work only approximates. This is the paper's most convincing evidence.
-
Unusually honest reporting. The authors disclose the limits of their own guarantee (Prop. 2) and report a failure case (Llama-3-8B cold start) with extended-horizon follow-up. This raises confidence in what is reported, though it does not resolve the decisive issues below.
Critical weaknesses
-
The proposed method fails to reduce training cost, contradicting its main motivation. To compute the ghost Gram matrix, the algorithm still needs a forward pass and per-sample backward signals on the full 2k candidate batch, plus another full forward and backward pass on the selected k subset. As shown in the paper's own runtime tables, a single GhostMatch step is 1.5 to 3 times slower than a standard full-batch step. However, the paper only compares methods at a fixed number of steps instead of fixed compute, and provides no wall-clock or FLOP curves. If evaluated at equal wall-clock time, the full-batch baseline would run for 2 to 3 times as many steps. In fact, Appendix F already shows that the full-batch baseline catches up when trained longer. Thus, there is no evidence that GhostMatch actually speeds up training.
-
Missing random baseline and weak statistical significance. The paper never compares against the simplest baseline of picking k examples uniformly at random. Without this control, we cannot know if the gains come from the selection metric or just from the reweighting mechanism. For instance, on Llama-2-7B, all five tested methods cluster tightly between 9.38 and 9.95 ppl, a range where random selection would easily fall. Moreover, the headline result in the abstract (0.609 vs. 0.572 F1 against OPUS) is averaged over only 2 runs with a standard deviation of 0.026, which is not statistically meaningful. Finally, the theoretical (1−1/e) guarantee only holds for the shifted facility-location objective, not for the actual representation error used in the convergence proof (which suffers from an extra error term in Prop. 2). As a result, the submodular approximation does not actually guarantee good convergence.
Questions for authors
-
Is there any configuration where GhostMatch reaches a given evaluation loss with less total compute (wall-clock or FLOPs, scoring pass included) than Regular full-batch training? If yes, please show compute-normalized curves; if no, what is the method's practical value?
-
Where does random-k selection (same k, uniform or cluster-reweighted) land in Table 1 for each configuration?
-
Can you provide any worst-case guarantee on the realized representation cost , which is the quantity that actually enters Theorem 1, beyond the additive bound in Proposition 2? If not, what does the guarantee actually imply for optimization?