- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes LPEC, a target-derived low-rank proxy for gradient-based data selection. A weighted decomposition supplies candidate components; estimates of forward and backward propagation determine which components to retain; and an interaction-aware procedure allocates ranks across matrices under a shared storage budget. A conditional first-order analysis motivates considering propagated compression errors. Experiments use three target models, three candidate datasets, and three downstream evaluation tasks. LPEC often improves over storage-matched IProX, including eight of nine model–dataset averages at sparsity 0.7. The implementation also introduces block-balanced factor-gradient cosine scoring, distinct from both the analytical alignment and the baseline scoring protocol.
Critical strengths
-
The method considers how a compression error changes activations and gradients at other matrices, which local influence-preserving compression does not account for. Both component ranking and rank allocation use these propagated effects. The paper also clearly states the limits of the analysis: it does not guarantee alignment preservation in the full Transformer.
-
The evaluation spans several model families, candidate datasets, and compression levels with matched factor-storage and selection budgets. Improvements over IProX are especially substantial on several Llama/Dolly settings, and the reported ablations support contributions from both propagation-aware ranking and joint allocation. These results suggest that the complete selection pipeline can be useful.
Critical weaknesses
-
The main comparison changes both compression and scoring, so it is unclear how much of the improvement comes from compression itself. Appendix A.1 states that every baseline uses joint cosine aggregation while LPEC uses block-balanced aggregation. Table 8 helps, but it covers only Dolly at sparsity 0.7, uses a different cap setting, and is a single run. On Qwen3 under joint aggregation, LPEC’s average is 61.70 versus IProX’s 61.97; changing the aggregation changes the comparison. The analysis concerns gradients with respect to the original weight matrices, whereas the deployed score uses gradients with respect to the low-rank factors. A direct comparison of the resulting candidate rankings would help connect the analysis to selection.
-
The reported timings do not show a saving for one round of data selection. LPEC takes approximately 25 minutes for construction and 76 for scoring, or 101 minutes in total, compared with approximately 100 minutes for target-direct scoring and 94 for IProX. Reusing the proxy could make it worthwhile, but this case is not measured. The cap is also swept separately for each model–dataset–sparsity setting without a clear validation selection rule, and the downstream results lack repeated-run variation. The total cost of obtaining the reported results is therefore unclear.
Questions for authors
-
Can you compare LPEC and IProX under the same joint aggregation and the same block-balanced aggregation, using the same cap-selection protocol, for representative settings from the main table? Do the compression gains persist?
-
How is the cap chosen after the per-setting sweep? Which validation data are used, and are the reported test scores excluded from this choice? What variation is observed across repeated probe sampling, selection, and fine-tuning runs?
-
What is the total time for construction, configuration selection, and scoring? Which parts of the proxy can be reused for later selection runs? How closely do candidate rankings under the deployed score agree with target-direct rankings?
3 comments
Response to Reviewer 9rKT (2)
2. Q2: Clarification of the Role of τ, Exclusion of Test Scores from Configuration, and Variation Across Repeated Runs
(a) Clarification of τ: a compression parameter on a par with s, reported in the main table rather than selected (revised: Appendix A.3 and Table 1; full sweep tabulated in this response). In the revised manuscript we clarify the role of τ: like the sparsity s, it is a compression parameter that bounds how far each matrix's retained ratio may deviate from the uniform storage ratio (Appendix A.3). The main table involves no τ sweep and no τ selection: we report LPEC results at τ ∈ {0.3, 0.5, 0.7} side by side for every s (Table 1), so the main-table values are not chosen at all and no validation split or metric determines a choice; the remaining experiments use τ = 0.3. On validation data: the only validation data used anywhere in the pipeline are the anchor sets (the MMLU/BBH/TyDiQA validation splits) that supply the target gradients for scoring, and τ itself is fitted to no data. We confirm that test scores are excluded from every configuration choice: they are computed once, on the finally selected subset, and in particular play no role in any choice involving τ. For completeness, the extended sweep over τ ∈ {0, 0.3, 0.5, 0.7, 1} is tabulated in full below:
| Model | Dataset | ||||||
|---|---|---|---|---|---|---|---|
| Qwen2.5-7B | Dolly | 0.3 | 56.86 | 57.74 | 57.19 | 57.33 | 57.92 |
| Qwen2.5-7B | Dolly | 0.5 | 57.83 | 58.64 | 58.47 | 57.72 | 57.59 |
| Qwen2.5-7B | Dolly | 0.7 | 57.83 | 57.16 | 58.54 | 57.68 | 57.00 |
| Qwen2.5-7B | Math | 0.3 | 52.10 | 52.85 | 52.45 | 52.78 | 52.70 |
| Qwen2.5-7B | Math | 0.5 | 52.26 | 52.53 | 52.40 | 51.80 | 52.21 |
| Qwen2.5-7B | Math | 0.7 | 52.07 | 52.89 | 52.02 | 51.93 | 51.66 |
| Qwen2.5-7B | Biomed | 0.3 | 53.13 | 55.18 | 54.41 | 54.28 | 54.12 |
| Qwen2.5-7B | Biomed | 0.5 | 55.19 | 56.30 | 54.81 | 54.26 | 53.61 |
| Qwen2.5-7B | Biomed | 0.7 | 55.17 | 56.57 | 53.85 | 54.19 | 55.56 |
| Qwen3-4B | Dolly | 0.3 | 61.88 | 61.09 | 61.60 | 61.79 | 61.90 |
| Qwen3-4B | Dolly | 0.5 | 62.19 | 61.68 | 61.76 | 60.98 | 61.64 |
| Qwen3-4B | Dolly | 0.7 | 62.06 | 63.19 | 62.27 | 61.73 | 62.25 |
| Qwen3-4B | Math | 0.3 | 60.91 | 60.89 | 61.07 | 60.98 | 60.99 |
| Qwen3-4B | Math | 0.5 | 61.00 | 61.21 | 61.15 | 61.23 | 61.33 |
| Qwen3-4B | Math | 0.7 | 60.55 | 61.33 | 60.83 | 60.72 | 60.57 |
| Qwen3-4B | Biomed | 0.3 | 62.86 | 62.97 | 62.99 | 63.06 | 62.96 |
| Qwen3-4B | Biomed | 0.5 | 62.74 | 63.23 | 62.99 | 62.97 | 63.35 |
| Qwen3-4B | Biomed | 0.7 | 62.96 | 63.54 | 62.34 | 62.60 | 63.18 |
| Llama3.2-3B | Dolly | 0.3 | 42.54 | 42.21 | 39.61 | 40.68 | 38.94 |
| Llama3.2-3B | Dolly | 0.5 | 41.91 | 42.05 | 39.92 | 40.96 | 40.41 |
| Llama3.2-3B | Dolly | 0.7 | 41.67 | 43.71 | 39.76 | 41.26 | 39.17 |
| Llama3.2-3B | Math | 0.3 | 42.80 | 43.08 | 42.48 | 42.78 | 42.68 |
| Llama3.2-3B | Math | 0.5 | 42.58 | 43.35 | 42.89 | 42.74 | 42.33 |
| Llama3.2-3B | Math | 0.7 | 42.60 | 42.83 | 42.58 | 42.93 | 43.16 |
| Llama3.2-3B | Biomed | 0.3 | 42.54 | 43.02 | 42.62 | 43.79 | 42.58 |
| Llama3.2-3B | Biomed | 0.5 | 45.48 | 44.75 | 43.36 | 42.76 | 42.90 |
| Llama3.2-3B | Biomed | 0.7 | 44.20 | 45.52 | 44.68 | 44.63 | 43.01 |
(b) Variation across repeated probe sampling, selection, and fine-tuning (probe diagnostics: Table 3; selection-stability). We measured the three sources of run-to-run variation the reviewer names. Probe sampling: the probe budget and composition diagnostics (Table 3) show that changing the probe budget by a factor of ten, or replacing up to 30% of the probes with interpolations, moves the downstream average by at most about 2.4 points. Selection: independent scoring runs under an identical configuration produce top-5% sets that are at least 99.7% identical. Fine-tuning: because only 5% of the candidate data is used for training, repeated fine-tuning runs of a fixed selection vary little.
Response to Reviewer 9rKT (3)
3. W2 & Q3: End-to-End Time Cost with Proxy Reuse, and Agreement Between Deployed and Target-Direct Selections
(a) End-to-end cost, configuration selection, and proxy reuse (revised: Table 4 and the Efficiency paragraph, Section 5.3). The revised Table 4 reports construction, scoring, and peak memory together for the dense target, IProX, and LPEC at three sparsities. Since τ is part of the compression configuration rather than a tuned knob (see our response to Q2), no sweep runs inside the pipeline and the end-to-end cost per sparsity is exactly construction plus scoring: about 101 minutes at s = 0.5 (on par with the ~100 minutes of target-direct scoring), and 88 and 77 minutes at s = 0.7 and s = 0.9, below both target-direct scoring and IProX (94 minutes). Proxy reuse compounds the advantage: construction is one-off and independent of the candidate pool (it uses 1% of the data source), and the low-rank factors, cached curvature cores, and token cache are all reusable, so a new pool or a new scoring round only recomputes the candidate scoring pass; two scoring rounds at s = 0.5 cost about 177 minutes against 200 for target-direct scoring. Peak memory follows the same trend: LPEC's scoring stage peaks at 39.8, 34.3, and 27.3 GiB at s = 0.5, 0.7, and 0.9, i.e., 0.90×, 0.78×, and 0.62× of the 44.0 GiB dense peak, so stronger compression lowers both time and memory.
| Method | Compress (min) | Infl. Calc. (min) | Peak (GiB) |
|---|---|---|---|
| Llama3.2-3B | -- | ~100 | 44.0 |
| IProX (, 1.8B) | ~19 | ~75 | 39.8 |
| LPEC (, 1.8B) | ~25 | ~76 | 39.8 |
| LPEC (, 1.2B) | ~25 | ~63 | 34.3 |
| LPEC (, 0.7B) | ~25 | ~52 | 27.3 |
(b) Agreement between the deployed ranking and target-direct selection (new: Table 10, Appendix C; supporting: Figure 3, Section 5.3). The revised manuscript measures this agreement directly at the level of the selected subsets: the gradient subspaces spanned by the LPEC top-5% selection and by the target's own selection are compared via subspace affinity, and the selection diversity is reported alongside. The overlap is moderate and stable across sparsities (SA 39.5–42.7), and the LPEC selections are highly diverse (1-NND 75.6–77.8):
| SA MMLU | SA BBH | SA TyDiQA | SA avg | 1-NND MMLU | 1-NND BBH | 1-NND TyDiQA | 1-NND avg | |
|---|---|---|---|---|---|---|---|---|
| 0.3 | 41.3 | 39.1 | 38.2 | 39.5 | 71.5 | 84.3 | 73.4 | 76.4 |
| 0.5 | 38.7 | 39.8 | 49.5 | 42.7 | 77.1 | 85.4 | 64.4 | 75.6 |
| 0.7 | 38.6 | 40.5 | 39.7 | 39.6 | 71.8 | 83.6 | 77.9 | 77.8 |
On the score side, Figure 3 (Section 5.3) shows that the signals the deployed score consumes remain close to the dense reference under the deployed factor gradients: the compressed proxy exhibits competitive, and in most settings lower, per-layer propagation error than IProX's whitened weighted SVD. Together, the two diagnostics connect the deployed ranking to the target-direct one at both the selection and the score level.
Response to Reviewer 9rKT (1)
We thank the reviewer for the careful review and the constructive comments. We address each point below, with the corresponding revisions of the manuscript indicated at the head of each point.
1. W1 & Q1: Joint vs. Block-Balanced Aggregation under an Identical Cap Protocol: Does the Compression Gain Persist?
(a) Full aggregation control under a unified protocol (expanded: Table 9, Appendix C). To eliminate the protocol confound, the revised manuscript adds experiments in which both methods are evaluated under a unified protocol: Table 9 reports LPEC and IProX under both the joint and the block-balanced aggregation over all three candidate datasets, both target models, and s ∈ {0.3, 0.5, 0.7}, with identical selection budget, factor-storage budget, and downstream training; only the scoring aggregation differs:
The table shows that the compression gain persists under both aggregations: LPEC leads IProX on the grid average under joint (52.46 vs. 51.53) and under block-balanced (52.76 vs. 51.64), winning 31 of the 36 cells; and it is IProX whose averages deteriorate under block-balanced aggregation (below its own joint average in 11 of 18 settings), while LPEC's remain stable — so the gain cannot be attributed to the aggregation choice.
(b) Connecting the analysis to the deployed ranking. The revised manuscript bridges the analysis in the original weight coordinates and the deployed factor-gradient score with two direct measurements: the propagation-error comparison of the deployed proxy against the dense reference (Figure 3, Section 5.3), and the selection-subspace fidelity of the deployed criterion against the target's own selection (Table 10, Appendix C). Both are discussed in our response to W2/Q3.