acceptodds
Under review as a conference paper at ICLR 2027

Layer-propagation Equivalence Compression for Gradient-Based Data Selection in LLM Fine-Tuning

Abstract

Gradient-based data selection identifies useful training examples by comparing their gradients with those of a target validation set, which is crucial to large language model (LLM) supervised fine-tuning (SFT). Computing these gradients with an LLM scales poorly, making it impractical for multi-billion-parameter models, motivating the construction of smaller proxy models through compression for gradient-influence equivalence to preserve the gradient information used for data selection. However, existing compression objectives evaluate each weight matrix separately, without accounting for how the resulting errors propagate across layers; we find that these errors can persist and become amplified, potentially altering the gradient alignment scores used for data selection. To address this issue, we introduce Layer-propagation Equivalence Compression (LPEC), a method that accounts for error propagation when constructing a low-rank compressed proxy. Specifically, we perform propagation-aware direction ranking by considering both singular values and how removal errors propagate to other layers. We then perform joint rank allocation across weight matrices under a shared parameter budget, giving more rank to matrices where retaining additional components is expected to reduce compression error more effectively. Experimental results across different LLM families and evaluation tasks demonstrate that LPEC improves downstream SFT performance over existing influence-preserving compression methods. Anonymous code is available at https://anonymous.4open.science/r/LEPC-119E/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Data Selection

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9rKTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes LPEC, a target-derived low-rank proxy for gradient-based data selection. A weighted decomposition supplies candidate components; estimates of forward and backward propagation determine which components to retain; and an interaction-aware procedure allocates ranks across matrices under a shared storage budget. A conditional first-order analysis motivates considering propagated compression errors. Experiments use three target models, three candidate datasets, and three downstream evaluation tasks. LPEC often improves over storage-matched IProX, including eight of nine model–dataset averages at sparsity 0.7. The implementation also introduces block-balanced factor-gradient cosine scoring, distinct from both the analytical alignment and the baseline scoring protocol.

Critical strengths
  1. The method considers how a compression error changes activations and gradients at other matrices, which local influence-preserving compression does not account for. Both component ranking and rank allocation use these propagated effects. The paper also clearly states the limits of the analysis: it does not guarantee alignment preservation in the full Transformer.

  2. The evaluation spans several model families, candidate datasets, and compression levels with matched factor-storage and selection budgets. Improvements over IProX are especially substantial on several Llama/Dolly settings, and the reported ablations support contributions from both propagation-aware ranking and joint allocation. These results suggest that the complete selection pipeline can be useful.

Critical weaknesses
  1. The main comparison changes both compression and scoring, so it is unclear how much of the improvement comes from compression itself. Appendix A.1 states that every baseline uses joint cosine aggregation while LPEC uses block-balanced aggregation. Table 8 helps, but it covers only Dolly at sparsity 0.7, uses a different cap setting, and is a single run. On Qwen3 under joint aggregation, LPEC’s average is 61.70 versus IProX’s 61.97; changing the aggregation changes the comparison. The analysis concerns gradients with respect to the original weight matrices, whereas the deployed score uses gradients with respect to the low-rank factors. A direct comparison of the resulting candidate rankings would help connect the analysis to selection.

  2. The reported timings do not show a saving for one round of data selection. LPEC takes approximately 25 minutes for construction and 76 for scoring, or 101 minutes in total, compared with approximately 100 minutes for target-direct scoring and 94 for IProX. Reusing the proxy could make it worthwhile, but this case is not measured. The cap is also swept separately for each model–dataset–sparsity setting without a clear validation selection rule, and the downstream results lack repeated-run variation. The total cost of obtaining the reported results is therefore unclear.

Questions for authors
  1. Can you compare LPEC and IProX under the same joint aggregation and the same block-balanced aggregation, using the same cap-selection protocol, for representative settings from the main table? Do the compression gains persist?

  2. How is the cap chosen after the per-setting sweep? Which validation data are used, and are the reported test scores excluded from this choice? What variation is observed across repeated probe sampling, selection, and fine-tuning runs?

  3. What is the total time for construction, configuration selection, and scoring? Which parts of the proxy can be reused for later selection runs? How closely do candidate rankings under the deployed score agree with target-direct rankings?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 9rKT (1)

We thank the reviewer for the careful review and the constructive comments. We address each point below, with the corresponding revisions of the manuscript indicated at the head of each point.

1. W1 & Q1: Joint vs. Block-Balanced Aggregation under an Identical Cap Protocol: Does the Compression Gain Persist?

(a) Full aggregation control under a unified protocol (expanded: Table 9, Appendix C). To eliminate the protocol confound, the revised manuscript adds experiments in which both methods are evaluated under a unified protocol: Table 9 reports LPEC and IProX under both the joint and the block-balanced aggregation over all three candidate datasets, both target models, and s ∈ {0.3, 0.5, 0.7}, with identical selection budget, factor-storage budget, and downstream training; only the scoring aggregation differs:

ModelDatasetssJoint LPECJoint IProXBlock-bal. LPECBlock-bal. IProXΔ LPECΔ IProX
Qwen3-4BDolly0.361.7361.1161.0961.47−0.64+0.36
Qwen3-4BDolly0.562.2761.4261.6860.74−0.59−0.68
Qwen3-4BDolly0.762.3762.9163.1960.62+0.82−2.29
Qwen3-4BMath0.360.8560.3960.8960.05+0.04−0.34
Qwen3-4BMath0.561.2160.6761.2160.280.00−0.39
Qwen3-4BMath0.760.8760.9261.3360.91+0.46−0.01
Qwen3-4BBiomed0.362.9862.8462.9762.82−0.01−0.02
Qwen3-4BBiomed0.563.3362.5863.2363.29−0.10+0.71
Qwen3-4BBiomed0.762.9962.4663.5462.42+0.55−0.04
Llama3.2-3BDolly0.343.6538.5542.2139.59−1.44+1.04
Llama3.2-3BDolly0.542.0739.2942.0540.93−0.02+1.64
Llama3.2-3BDolly0.740.1837.9443.7141.74+3.53+3.80
Llama3.2-3BMath0.342.7042.4943.0842.40+0.38−0.09
Llama3.2-3BMath0.542.4542.3443.3542.46+0.90+0.12
Llama3.2-3BMath0.742.7042.6742.8342.57+0.13−0.10
Llama3.2-3BBiomed0.342.8443.6243.0242.07+0.18−1.55
Llama3.2-3BBiomed0.544.7243.3144.7541.73+0.03−1.58
Llama3.2-3BBiomed0.744.4342.1045.5243.39+1.09+1.29

The table shows that the compression gain persists under both aggregations: LPEC leads IProX on the grid average under joint (52.46 vs. 51.53) and under block-balanced (52.76 vs. 51.64), winning 31 of the 36 cells; and it is IProX whose averages deteriorate under block-balanced aggregation (below its own joint average in 11 of 18 settings), while LPEC's remain stable — so the gain cannot be attributed to the aggregation choice.

(b) Connecting the analysis to the deployed ranking. The revised manuscript bridges the analysis in the original weight coordinates and the deployed factor-gradient score with two direct measurements: the propagation-error comparison of the deployed proxy against the dense reference (Figure 3, Section 5.3), and the selection-subspace fidelity of the deployed criterion against the target's own selection (Table 10, Appendix C). Both are discussed in our response to W2/Q3.

AuthorsOpenReview ↗

Response to Reviewer 9rKT (2)

2. Q2: Clarification of the Role of τ, Exclusion of Test Scores from Configuration, and Variation Across Repeated Runs

(a) Clarification of τ: a compression parameter on a par with s, reported in the main table rather than selected (revised: Appendix A.3 and Table 1; full sweep tabulated in this response). In the revised manuscript we clarify the role of τ: like the sparsity s, it is a compression parameter that bounds how far each matrix's retained ratio may deviate from the uniform storage ratio (Appendix A.3). The main table involves no τ sweep and no τ selection: we report LPEC results at τ ∈ {0.3, 0.5, 0.7} side by side for every s (Table 1), so the main-table values are not chosen at all and no validation split or metric determines a choice; the remaining experiments use τ = 0.3. On validation data: the only validation data used anywhere in the pipeline are the anchor sets (the MMLU/BBH/TyDiQA validation splits) that supply the target gradients for scoring, and τ itself is fitted to no data. We confirm that test scores are excluded from every configuration choice: they are computed once, on the finally selected subset, and in particular play no role in any choice involving τ. For completeness, the extended sweep over τ ∈ {0, 0.3, 0.5, 0.7, 1} is tabulated in full below:

ModelDatasetssτ=0\tau{=}0τ=0.3\tau{=}0.3τ=0.5\tau{=}0.5τ=0.7\tau{=}0.7τ=1\tau{=}1
Qwen2.5-7BDolly0.356.8657.7457.1957.3357.92
Qwen2.5-7BDolly0.557.8358.6458.4757.7257.59
Qwen2.5-7BDolly0.757.8357.1658.5457.6857.00
Qwen2.5-7BMath0.352.1052.8552.4552.7852.70
Qwen2.5-7BMath0.552.2652.5352.4051.8052.21
Qwen2.5-7BMath0.752.0752.8952.0251.9351.66
Qwen2.5-7BBiomed0.353.1355.1854.4154.2854.12
Qwen2.5-7BBiomed0.555.1956.3054.8154.2653.61
Qwen2.5-7BBiomed0.755.1756.5753.8554.1955.56
Qwen3-4BDolly0.361.8861.0961.6061.7961.90
Qwen3-4BDolly0.562.1961.6861.7660.9861.64
Qwen3-4BDolly0.762.0663.1962.2761.7362.25
Qwen3-4BMath0.360.9160.8961.0760.9860.99
Qwen3-4BMath0.561.0061.2161.1561.2361.33
Qwen3-4BMath0.760.5561.3360.8360.7260.57
Qwen3-4BBiomed0.362.8662.9762.9963.0662.96
Qwen3-4BBiomed0.562.7463.2362.9962.9763.35
Qwen3-4BBiomed0.762.9663.5462.3462.6063.18
Llama3.2-3BDolly0.342.5442.2139.6140.6838.94
Llama3.2-3BDolly0.541.9142.0539.9240.9640.41
Llama3.2-3BDolly0.741.6743.7139.7641.2639.17
Llama3.2-3BMath0.342.8043.0842.4842.7842.68
Llama3.2-3BMath0.542.5843.3542.8942.7442.33
Llama3.2-3BMath0.742.6042.8342.5842.9343.16
Llama3.2-3BBiomed0.342.5443.0242.6243.7942.58
Llama3.2-3BBiomed0.545.4844.7543.3642.7642.90
Llama3.2-3BBiomed0.744.2045.5244.6844.6343.01

(b) Variation across repeated probe sampling, selection, and fine-tuning (probe diagnostics: Table 3; selection-stability). We measured the three sources of run-to-run variation the reviewer names. Probe sampling: the probe budget and composition diagnostics (Table 3) show that changing the probe budget by a factor of ten, or replacing up to 30% of the probes with interpolations, moves the downstream average by at most about 2.4 points. Selection: independent scoring runs under an identical configuration produce top-5% sets that are at least 99.7% identical. Fine-tuning: because only 5% of the candidate data is used for training, repeated fine-tuning runs of a fixed selection vary little.

AuthorsOpenReview ↗

Response to Reviewer 9rKT (3)

3. W2 & Q3: End-to-End Time Cost with Proxy Reuse, and Agreement Between Deployed and Target-Direct Selections

(a) End-to-end cost, configuration selection, and proxy reuse (revised: Table 4 and the Efficiency paragraph, Section 5.3). The revised Table 4 reports construction, scoring, and peak memory together for the dense target, IProX, and LPEC at three sparsities. Since τ is part of the compression configuration rather than a tuned knob (see our response to Q2), no sweep runs inside the pipeline and the end-to-end cost per sparsity is exactly construction plus scoring: about 101 minutes at s = 0.5 (on par with the ~100 minutes of target-direct scoring), and 88 and 77 minutes at s = 0.7 and s = 0.9, below both target-direct scoring and IProX (94 minutes). Proxy reuse compounds the advantage: construction is one-off and independent of the candidate pool (it uses 1% of the data source), and the low-rank factors, cached curvature cores, and token cache are all reusable, so a new pool or a new scoring round only recomputes the candidate scoring pass; two scoring rounds at s = 0.5 cost about 177 minutes against 200 for target-direct scoring. Peak memory follows the same trend: LPEC's scoring stage peaks at 39.8, 34.3, and 27.3 GiB at s = 0.5, 0.7, and 0.9, i.e., 0.90×, 0.78×, and 0.62× of the 44.0 GiB dense peak, so stronger compression lowers both time and memory.

MethodCompress (min)Infl. Calc. (min)Peak (GiB)
Llama3.2-3B--~10044.0
IProX (s=0.5s{=}0.5, 1.8B)~19~7539.8
LPEC (s=0.5s{=}0.5, 1.8B)~25~7639.8
LPEC (s=0.7s{=}0.7, 1.2B)~25~6334.3
LPEC (s=0.9s{=}0.9, 0.7B)~25~5227.3

(b) Agreement between the deployed ranking and target-direct selection (new: Table 10, Appendix C; supporting: Figure 3, Section 5.3). The revised manuscript measures this agreement directly at the level of the selected subsets: the gradient subspaces spanned by the LPEC top-5% selection and by the target's own selection are compared via subspace affinity, and the selection diversity is reported alongside. The overlap is moderate and stable across sparsities (SA 39.5–42.7), and the LPEC selections are highly diverse (1-NND 75.6–77.8):

ssSA MMLUSA BBHSA TyDiQASA avg1-NND MMLU1-NND BBH1-NND TyDiQA1-NND avg
0.341.339.138.239.571.584.373.476.4
0.538.739.849.542.777.185.464.475.6
0.738.640.539.739.671.883.677.977.8

On the score side, Figure 3 (Section 5.3) shows that the signals the deployed score consumes remain close to the dense reference under the deployed factor gradients: the compressed proxy exhibits competitive, and in most settings lower, per-layer propagation error than IProX's whitened weighted SVD. Together, the two diagnostics connect the deployed ranking to the target-direct one at both the selection and the score level.

Reviewer_qQLFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on building a better proxy model for estimating gradient-based influence in LLM fine-tuning. The proposed LPEC builds on prior work in two ways: it considers how compression errors propagate across layers and jointly allocates rank to further reduce their overall impact. It uses forward and backward propagation estimates to rerank low-rank components, then considers interactions between compression errors given a fixed storage budget. The paper provides sound theoretical motivation and reports improvements over IProX in most evaluation settings.

Critical strengths
  1. The problem is clearly defined and well-motivated. The svd-style compression typically only considers inner-layer effect, and methods like IPROX need an additional alignment stage to correct the resulting error. This paper directly tackles this issue by considering the cross-layer error propagation.

  2. The theoretical analysis provides a clear view and useful motivation for the whole method.

  3. The evaluation covers different models and benchmarks. The reported results improve over baselines in most settings, and further analysis is provided to better understand the method.

Critical weaknesses
  1. The scope is relatively focused on single gradient-based influence approximation, and the method mainly developed in the weighted-SVD framework. The setting is well motivated, but broader use remains unexplored. Need more discussion here.

  2. The results show downstream gains, but do not directly measure how much LPEC reduces propagated error compared with weighted SVD (which is one important claim of the paper). Both small simulation in synthetic setting or direct error comparison on real data would help.

Questions for authors
  1. Does the IPROX baseline include Stage 2 gradient alignment?

  2. Table 4 shows little overall time saving (compression + influence calculation) for both IPROX and LPEC. Where does the efficiency gain mainly come from?

  3. Can LPEC extend to influence functions?

2 comments
AuthorsOpenReview ↗

Response to Reviewer qQLF (1)

We thank the reviewer for the careful review and the constructive comments. We address each point below, with the corresponding revisions of the manuscript indicated at the head of each point.

1. Q1: Completeness of the Reproduced IProX Baseline (Inclusion of the Stage-2 Gradient Alignment) (clarified: Baselines, Appendix A.1)

Yes. Our reproduction runs the official IProX implementation in full, i.e., both stages of the original method: the influence-preserving compression stage (layer-local whitened weighted SVD) and the stage-2 proxy–target gradient/logit alignment on the official probe split. Every comparison in the paper is therefore against the complete method, not against an IPSVD-initialization-only variant. This is now stated explicitly in the Baselines appendix of the revised manuscript.


2. Q2: Where the Efficiency Gain Comes From Beyond Single-Round Wall-Clock Time (revised: Table 4 and the Efficiency paragraph, Section 5.3)

At s = 0.5 a single selection round costs about 101 minutes end-to-end for LPEC (25 construction + 76 scoring), on par with target-direct scoring; the efficiency gain of the compressed proxy lies elsewhere, and the revised Table 4 and Efficiency paragraph (Section 5.3) make each source explicit:

  • Compression rate. Scoring cost scales with the retained rank, not with the target model: it drops from 76 to 63 and 52 minutes at s = 0.7 and s = 0.9, so the end-to-end cost per round falls to 88 and 77 minutes — below both target-direct scoring (~100 min) and IProX (94 min) — while selection quality is maintained (Table 2).
  • Reuse. The compressed proxy is reusable: once constructed, it scores any candidate pool or repeated scoring round without reconstruction, amortizing the one-off construction over all subsequent selection runs.
  • Memory. The scoring-stage peak GPU memory falls to 39.8 / 34.3 / 27.3 GiB at s = 0.5 / 0.7 / 0.9, i.e., 0.90 / 0.78 / 0.62 times the dense target's 44.0 GiB — an efficiency axis on which the compressed proxy improves monotonically where wall-clock alone does not.
MethodCompress (min)Infl. Calc. (min)Peak (GiB)
Llama3.2-3B--~10044.0
IProX (s=0.5s{=}0.5, 1.8B)~19~7539.8
LPEC (s=0.5s{=}0.5, 1.8B)~25~7639.8
LPEC (s=0.7s{=}0.7, 1.2B)~25~6334.3
LPEC (s=0.9s{=}0.9, 0.7B)~25~5227.3

AuthorsOpenReview ↗

Response to Reviewer qQLF (2)

3. W1 & Q3: Extension to Influence Functions and the Broader Applicability of the Framework (new: Appendix K; Table 5, Section 5.3)

Yes, LPEC extends to influence functions, and the extension is new in the revised manuscript. The per-layer second moments Ch,kC_{h,k} and CΔ,kC_{\Delta,k} that our propagation bound already estimates are exactly the two Kronecker factors of K-FAC, because each layer's weight gradient is a rank-one outer product. Promoting them from bound coefficients to preconditioners, and inverting them only inside the retained rank-rkr_k subspaces, turns the first-order alignment into a layer-structured, EK-FAC-style influence function

IF(z,z′)=−∑k=1L(hk−1(z′),Ak−1hk−1(z))(δk(z′),Bk−1δk(z))\mathrm{IF}(z, z^{\prime}) = -\sum_{k=1}^{L} (h_{k-1}(z^{\prime}), A_{k}^{-1} h_{k-1}(z)) (\delta_{k}(z^{\prime}), B_{k}^{-1} \delta_{k}(z))

where hk−1h_{k-1} and δk\delta_{k} are the layer input activation and the back-propagated error, and AkA_{k} and BkB_{k} are the damped Kronecker factors (written h^\hat{h}, δ^\hat{\delta}, A~\tilde{A}, B~\tilde{B} in the notation of Appendix K). The score reduces to the deployed first-order score when the preconditioners are the identity, and its inverse costs O(rk3)O(r_k^3) once per layer, cheaper than a dense gradient inner product, i.e., it is the compression itself that makes the curvature term affordable. Table 5 (Section 5.3) confirms that selecting with this influence-function score preserves downstream performance:

SelectorMMLUBBHTyDiQAAvg.
Llama3.2-3B52.3444.9026.7941.34
Llama3.2-1B51.8147.2927.7342.27
LPEC, s=0.3s{=}0.351.0546.9930.1842.74
LPEC, s=0.5s{=}0.551.7047.2931.0943.36
LPEC, s=0.7s{=}0.751.3247.2031.5143.34

The compressed-proxy influence-function selector matches the first-order production score (43.4 vs. 43.4) and stays stable across sparsities.

On the broader scope (W1): the framework is not tied to a single influence approximation. The compression construction consumes only second moments and propagation gains rather than a particular alignment score, and the scoring rule enters only through per-layer gradients, so any first-order selection score (multi-checkpoint TracIn sums, curvature-weighted influence functions as above) can be evaluated on the same proxy.


4. W2: Direct Measurement of Propagated-Error Reduction Against Weighted SVD (new: "Propagation error" paragraph and Figure 3, Section 5.3)

The revision adds a direct measurement of the propagated error of the compressed proxy against the weighted SVD, on the same target models and candidate datasets as the main experiments. The compressed layers of each arm are installed into the warmed-up target model, and for a set of candidates the squared relative output error against the dense reference is recorded per layer and summarized by the mean over layers, comparing LPEC with the whitened weighted SVD at matched per-layer rank budgets for s ∈ {0.3, 0.5, 0.7} on both target models (Figure 3 and the "Propagation error" paragraph, Section 5.3). LPEC attains the lower mean error for Qwen3-4B at every rate (by 1–4%), and is lower on Llama3.2-3B at s = 0.5 (about 6%), while weighted SVD is about 9% lower at s = 0.3 and s = 0.7. This connects the propagated-error claim of the analysis to a direct measurement, complementing the downstream results.

Reviewer_nExSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to compress an LLM into a smaller proxy for gradient-based data selection. LPEC starts from a weighted low-rank decomposition, reranks components using their estimated forward and backward error propagation, and jointly allocates ranks across matrices under a shared parameter budget while accounting for empirical error interactions. The paper provides a conditional first-order analysis for a simplified sequential model and evaluates the method across three target models, three candidate datasets, and three downstream benchmarks.

Critical strengths

The motivation is sound and the method addresses a concrete limitation. When multiple matrices are compressed simultaneously, minimizing their individual reconstruction errors need not preserve the network’s gradient behavior. Combining direction-specific propagation estimates with cross-matrix rank allocation is a reasonable extension of layer-local compression. The paper also explicitly distinguishes its conditional theoretical results from the heuristic approximations used in practice.

The experiments cover multiple models and data sources and demonstrate meaningful downstream gains. At sparsity (s=0.7), LPEC outperforms budget-matched IProX on eight of the nine model–candidate-dataset combinations. For example, on Llama3.2-3B, the average score improves from 37.94 to 43.71 on Dolly and from 42.10 to 45.52 on Biomed. Table 5 additionally ablates direction ranking, weighted SVD, interactions, and joint rank allocation, providing empirical support for these design choices.

Critical weaknesses

The main experiments do not sufficiently isolate the contribution of compression. Appendices A.1 and B specify that all baselines use a joint cosine over concatenated gradients, whereas LPEC averages separately normalized block-level cosines. This directly changes candidate rankings. The control in Table 8 is useful, but covers only Dolly, one sparsity, and two models, with a single run. It already shows that aggregation affects the relative conclusions: on Qwen3-4B, LPEC scores 61.70 versus IProX’s 61.97 under joint aggregation, but 62.06 versus 61.46 under block-balanced aggregation. Consequently, the main-table differences cannot be attributed entirely to propagation-aware compression. The absence of uncertainty estimates across repeated runs also makes the smaller improvements and ablation differences difficult to assess.

There remains a substantial gap between the analysis and the deployed selection criterion. Proposition 4.1 concerns unnormalized gradient inner products in the original weight-matrix coordinates, under a sequential model with fixed layer derivatives and terminal loss gradients. Deployment instead uses gradients of low-rank factor parameters followed by block-balanced cosine aggregation, introducing a different parameterization and normalization. Component ranking and allocation further use separability, omitted cross-terms, and simplified residual-error measures. The authors acknowledge these limitations, so I do not interpret the paper as claiming an invalid guarantee. However, downstream SFT scores alone do not directly establish whether these approximations better preserve the target model’s selection signal. Direct measurements of score distortion, ranking correlation, or top-5% selection overlap are particularly important under extreme compression.

Questions for authors

Can the authors provide the full LPEC/IProX × joint/block-balanced aggregation comparison across representative main settings, with matched tuning budgets? Does the IProX baseline include its original second-stage gradient and logit alignment, or only IPSVD initialization? This distinction matters for interpreting the comparison. Original IProX paper

Appendix A.3 states that (\tau) is swept for each model–dataset–sparsity setting. How are the final main-table values chosen, and which validation split and metric determine that choice? Can the authors confirm that test results are not used for selection and provide multiple-seed results together with a random 5% selection baseline?

For actual Transformers and deployed factor-gradient scores, does LPEC improve ranking correlation or top-5% overlap with the target model, or reduce score distortion relative to IProX? Can these diagnostics cover moderate and extreme compression and separate compression effects from aggregation effects?

Table 4 gives approximately (25+76=101) minutes for LPEC construction and scoring, versus approximately 100 minutes for direct target-model scoring. Can the authors report end-to-end costs including construction and parameter sweeps, together with amortized benefits for larger candidate pools or repeated scoring and peak memory usage?

5 comments
AuthorsOpenReview ↗

Response to Reviewer nExS(1)

We thank the reviewer for the careful review and the constructive comments. We address each point below, with the corresponding revisions of the manuscript indicated at the head of each point.

1. W1 & Q1: LPEC/IProX × Joint/Block-Balanced Aggregation Control Across Main Settings & Clarification of the IProX Baseline

(a) The aggregation control now covers all main comparison settings (expanded: Table 9, Appendix C). To eliminate the protocol confound, the revised manuscript adds experiments in which both methods are evaluated under a unified protocol: Table 9 reports LPEC and IProX under both the joint and the block-balanced aggregation over all three candidate datasets, both target models, and s∈{0.3,0.5,0.7}s\in\{0.3,0.5,0.7\}, with identical selection budget, factor-storage budget, and downstream training; only the scoring aggregation differs:

ModelDatasetssJoint LPECJoint IProXBlock-bal. LPECBlock-bal. IProXΔ LPECΔ IProX
Qwen3-4BDolly0.361.7361.1161.0961.47−0.64+0.36
Qwen3-4BDolly0.562.2761.4261.6860.74−0.59−0.68
Qwen3-4BDolly0.762.3762.9163.1960.62+0.82−2.29
Qwen3-4BMath0.360.8560.3960.8960.05+0.04−0.34
Qwen3-4BMath0.561.2160.6761.2160.280.00−0.39
Qwen3-4BMath0.760.8760.9261.3360.91+0.46−0.01
Qwen3-4BBiomed0.362.9862.8462.9762.82−0.01−0.02
Qwen3-4BBiomed0.563.3362.5863.2363.29−0.10+0.71
Qwen3-4BBiomed0.762.9962.4663.5462.42+0.55−0.04
Llama3.2-3BDolly0.343.6538.5542.2139.59−1.44+1.04
Llama3.2-3BDolly0.542.0739.2942.0540.93−0.02+1.64
Llama3.2-3BDolly0.740.1837.9443.7141.74+3.53+3.80
Llama3.2-3BMath0.342.7042.4943.0842.40+0.38−0.09
Llama3.2-3BMath0.542.4542.3443.3542.46+0.90+0.12
Llama3.2-3BMath0.742.7042.6742.8342.57+0.13−0.10
Llama3.2-3BBiomed0.342.8443.6243.0242.07+0.18−1.55
Llama3.2-3BBiomed0.544.7243.3144.7541.73+0.03−1.58
Llama3.2-3BBiomed0.744.4342.1045.5243.39+1.09+1.29

Under both aggregation protocols, LPEC outperforms IProX on average over the full grid (52.46 vs. 51.53 under joint; 52.76 vs. 51.64 under block-balanced). IProX performs markedly worse under block-balanced aggregation than under its native joint protocol, whereas LPEC's grid average is comparable under both, so the main-table ordering cannot be attributed to the aggregation choice.

(b) The IProX baseline is the complete algorithm, not IPSVD initialization alone (clarified: Baselines, Appendix A.1). Our reproduction runs the official IProX implementation in full, i.e., both stages of the original method: the influence-preserving compression stage (layer-local whitened weighted SVD) and the subsequent proxy–target alignment stage (gradient/logit alignment on the official probe split). This is now stated explicitly in the Baselines appendix (Appendix A.1) of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer nExS(2)

2. Q2: Clarification of the Role of τ, and the Random-5% Selection Baseline with Multiple Seeds

(a) Clarification of the role of τ: a compression parameter on a par with s, reported in the main table rather than selected (revised: Appendix A.3 and Table 1; full sweep tabulated in this response). In the revised manuscript we clarify the role of τ: like the sparsity s, it is a compression parameter that bounds how far each matrix's retained ratio may deviate from the uniform storage ratio (Appendix A.3). The main table involves no τ sweep and no τ selection: we report LPEC results at τ ∈ {0.3, 0.5, 0.7} side by side for every s (Table 1), which directly answers how the main-table values are chosen — they are not chosen at all, since every setting is shown, and no validation split or metric determines a choice. The remaining experiments use τ = 0.3. We also confirm that test results were never used for selection at any stage: candidate scoring consumes only the validation anchor sets (the MMLU/BBH/TyDiQA validation splits that supply the target gradients), and no configuration, including τ, was picked by any downstream test metric. For completeness, the extended sweep over τ ∈ {0, 0.3, 0.5, 0.7, 1} is tabulated in full below.

ModelDatasetssτ=0\tau{=}0τ=0.3\tau{=}0.3τ=0.5\tau{=}0.5τ=0.7\tau{=}0.7τ=1\tau{=}1
Qwen2.5-7BDolly0.356.8657.7457.1957.3357.92
Qwen2.5-7BDolly0.557.8358.6458.4757.7257.59
Qwen2.5-7BDolly0.757.8357.1658.5457.6857.00
Qwen2.5-7BMath0.352.1052.8552.4552.7852.70
Qwen2.5-7BMath0.552.2652.5352.4051.8052.21
Qwen2.5-7BMath0.752.0752.8952.0251.9351.66
Qwen2.5-7BBiomed0.353.1355.1854.4154.2854.12
Qwen2.5-7BBiomed0.555.1956.3054.8154.2653.61
Qwen2.5-7BBiomed0.755.1756.5753.8554.1955.56
Qwen3-4BDolly0.361.8861.0961.6061.7961.90
Qwen3-4BDolly0.562.1961.6861.7660.9861.64
Qwen3-4BDolly0.762.0663.1962.2761.7362.25
Qwen3-4BMath0.360.9160.8961.0760.9860.99
Qwen3-4BMath0.561.0061.2161.1561.2361.33
Qwen3-4BMath0.760.5561.3360.8360.7260.57
Qwen3-4BBiomed0.362.8662.9762.9963.0662.96
Qwen3-4BBiomed0.562.7463.2362.9962.9763.35
Qwen3-4BBiomed0.762.9663.5462.3462.6063.18
Llama3.2-3BDolly0.342.5442.2139.6140.6838.94
Llama3.2-3BDolly0.541.9142.0539.9240.9640.41
Llama3.2-3BDolly0.741.6743.7139.7641.2639.17
Llama3.2-3BMath0.342.8043.0842.4842.7842.68
Llama3.2-3BMath0.542.5843.3542.8942.7442.33
Llama3.2-3BMath0.742.6042.8342.5842.9343.16
Llama3.2-3BBiomed0.342.5443.0242.6243.7942.58
Llama3.2-3BBiomed0.545.4844.7543.3642.7642.90
Llama3.2-3BBiomed0.744.2045.5244.6844.6343.01

(b) The random-5% selection baseline with multiple seeds. We add a random-5% selection baseline run with three selection seeds (42 / 1024 / 2026), drawn from the same candidate pools under the same 5% budget and the same downstream protocol as the main table. The table below compares it with LPEC at all three sparsities (Δ = LPEC minus random-5% mean).

ModelDatasetRandom 5% (3-seed mean)seed rangeΔ s=0.3s{=}0.3Δ s=0.5s{=}0.5Δ s=0.7s{=}0.7
Qwen2.5-7BDolly56.131.45+1.61+2.51+1.03
Qwen2.5-7BMath51.901.46+0.95+0.63+0.99
Qwen2.5-7BBiomed54.931.75+0.25+1.37+1.64
Qwen3-4BDolly62.270.56−1.18−0.59+0.92
Qwen3-4BMath60.960.05−0.07+0.25+0.37
Qwen3-4BBiomed63.100.58−0.13+0.13+0.44
Llama3.2-3BDolly42.372.42−0.16−0.32+1.34
Llama3.2-3BMath42.820.26+0.26+0.53+0.01
Llama3.2-3BBiomed43.481.25−0.46+1.27+2.04

On Qwen2.5-7B, LPEC exceeds the random baseline in all nine model–dataset–rate cells (mean +1.22); on Qwen3-4B and Llama3.2-3B the margin over random grows monotonically with compression (−0.29 / +0.21 / +0.85 at s = 0.3 / 0.5 / 0.7, positive in all six cells at s = 0.7), even though the random baseline's own seed variability (ranges 0.05–2.42) forms a substantial noise floor at this small budget. Beyond this no-information reference, the controlled same-seed, same-basis comparison against IProX (Tables 1 and 9) and the mechanistic diagnostics (Figure 3, Figure 4, Table 10) provide the primary evidence for the method.

AuthorsOpenReview ↗

Response to Reviewer nExS(3)

3. W2 & Q3: Direct Diagnostics of Selection-Signal Preservation under Compression

To connect the analysis with the deployed criterion, the revision adds two direct diagnostics: a propagation-error measurement of the deployed compressed proxy, and a selection-fidelity measurement against the target's own selection. Together with the aggregation control of Table 9, they separate the compression effect from the aggregation effect.

(a) Propagation error of the deployed proxy (new: "Propagation error" paragraph and Figure 3, Section 5.3). This diagnostic is new in the revised manuscript (Section 5.3, "Propagation error" and Figure 3): it measures the per-layer squared relative output error of the compressed proxy against the dense reference, summarized by the mean over layers, in comparison with IProX's whitened weighted SVD at matched per-layer rank budgets. On Qwen3-4B, LPEC is lower at all three rates (by 1–4%); on Llama3.2-3B, LPEC is lower at s = 0.5 (about 6%) while the weighted SVD is about 9% lower at s = 0.3 and 0.7.

(b) Preservation of the target's selection signal (new: "Selection fidelity" subsection and Table 10, Appendix C). To measure the deployed criterion end-to-end, we compare the subset selected by LPEC with the subset the target model selects for itself (top 5% per task): using the same warmed-up target, we recompute per-example gradients for both subsets and compare the subspaces they span, via subspace affinity (SA, the mean over layers of the squared cosines of the principal angles) together with the selection diversity 1-NND. We use subspace affinity rather than raw set overlap because the gradients of the selected examples are the object that enters scoring: SA measures the functional content carried by a selection, whereas two sets can share few examples yet span nearly the same gradient subspace. The results (Table 10 of the revised manuscript, reproduced below) show a moderate, rate-stable overlap with the target's own selection and high, rate-flat diversity:

ssSA MMLUSA BBHSA TyDiQASA avg1-NND MMLU1-NND BBH1-NND TyDiQA1-NND avg
0.341.339.138.239.571.584.373.476.4
0.538.739.849.542.777.185.464.475.6
0.738.640.539.739.671.883.677.977.8

On the downstream side, the IProX-related concern is answered directly by the aggregation control: LPEC scores above IProX under both scoring protocols, joint cosine and block-balanced (mean Avg 52.46 vs. 51.53 and 52.76 vs. 51.64 across the 18 settings of Table 9), so the selection signal preserved by the deployed criterion translates into downstream gains over IProX regardless of the aggregation used. At extreme compression (s = 0.9, 0.99), LPEC likewise outperforms rate-matched IProX downstream in every cell (Table 2).

(c) Separating compression from aggregation (revised/new: Table 9, Appendix C; Figure 3, Section 5.3; Table 10, Appendix C). The two axes the reviewer asks to disentangle are crossed in separate controls: Table 9 varies only the scoring aggregation (identical selection pipeline and budgets), Figure 3 varies only the compression construction (matched rank budgets, and the layer-output error involves no scoring aggregation at all), and Table 10 evaluates the fully deployed criterion (factor gradients with block-balanced aggregation) against the target's own selection. Compression effects, aggregation effects, and end-to-end selection fidelity are therefore identified separately.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers and the AC

Summary of Rebuttal and Manuscript Revisions

We thank all reviewers for their careful and constructive feedback. The experiments and manuscript revisions we made during the discussion period to address the reviewers' concerns are summarized in the following points.

(1) Unified-protocol scoring-aggregation comparison (new Table 9, Appendix C; nExS W1/Q1, 9rKT W1/Q1). We expanded the aggregation control from a single setting to all three candidate datasets, both target models, and s∈{0.3,0.5,0.7}s \in \{0.3,0.5,0.7\} under both joint and block-balanced scoring, with identical budgets and downstream training. LPEC leads IProX on the grid average under both protocols so the gain is not an artifact of the aggregation choice.

(2) Clarification of the IProX baseline (revised Appendix A.1; nExS Q1, qQLF Q1). We now state explicitly that our reproduction runs the full official two-stage IProX pipeline, including the proxy–target alignment stage.

(3) Cap protocol clarified (revised Table 1, Section 5, Appendix A.3; nExS Q2, 9rKT Q2). τ\tau is a compression parameter on a par with ss: the main table reports τ∈{0.3,0.5,0.7}\tau \in \{0.3,0.5,0.7\} for every ss, all other experiments use the default τ=0.3\tau=0.3, and no validation or test metric enters any configuration choice. The full 27-setting sweep is reported in our individual responses.

(4) Direct diagnostics of selection-signal preservation (new Figure 3, Section 5.3; new Table 10, Appendix C; nExS W2/Q3, qQLF W2, 9rKT W2/Q3). We added a propagation-error comparison against weighted SVD at matched per-layer rank budgets, and a selection-fidelity diagnostic (subspace affinity and diversity) against the target model's own top-5% selection. Together with Table 9, compression, aggregation, and end-to-end fidelity are identified separately.

(5) End-to-end efficiency (revised Table 4 and Efficiency paragraph, Section 5.3; nExS Q4, qQLF Q2, 9rKT W2/Q3). Construction, scoring, and peak GPU memory are now reported together at s=0.5/0.7/0.9s=0.5/0.7/0.9: scoring drops to 63/52 minutes and peak memory to 0.78×/0.62×0.78\times/0.62\times of the dense target, and the one-off construction amortizes over repeated scoring rounds and new candidate pools.

(6) Influence-function extension (new Appendix K; new Table 5, Section 5.3; qQLF W1/Q3). LPEC is not tied to a single selection score: the framework extends to influence functions, reusing quantities already estimated during proxy construction at no additional estimation cost. Under this influence-function score, downstream performance is preserved across sparsities.

(7) Random-5% selection baseline (nExS Q2). We added a random-5% selection baseline run with three selection seeds.

We hope these additions address the reviewers' concerns and are happy to provide further clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.