acceptodds
Under review as a conference paper at ICLR 2027

DimPO: Dimensionality Reduction for Attention using Preference Optimization

Abstract

A linear projection might reduce the dimension of the query and key vectors in attention without updating the pretrained model, but it remains unclear which training objective best preserves the behavior of that model. The straightforward objective is to match the full attention distribution, for example by minimizing the KL divergence. We instead ask whether a preference over keys relative to a query, and preserving the attention mass of only the highest-weighted keys, provides a better training signal, especially in long-context settings. Existing reference-free preference objectives are mostly pairwise and underuse the full key set that listwise and KL supervision already use. We therefore introduce DimPO, a loss specific to this projection, which combines listwise preference optimization with no reference model and a lightweight top-k cross-entropy term for head-fidelity supervision. To verify the direct effect of our loss on how faithfully attention is preserved, DimPO is trained offline from the attention patterns of a frozen language model, with the query and key projections optimized independently per layer. Across LLaMA3.2-3B, LLaMA3.1-8B, Qwen2.5-7B, and Qwen3-4B Instruct models, pairwise preference objectives outperform the triplet objective baseline and, on short-context tasks, retain 98% of the original score when a projection to half the dimension is applied to at most the last 40% of the attention layers. When applied to more layers or evaluated on long-context RULER, however, they no longer maintain a reasonable performance. In contrast, KL and DimPO, which use every key during training, still retain about 95% of the original RULER 4k score on the 8B model when applied to up to 50% of the layers. KL-based projections nevertheless remain closer to the original attention distribution in terms of KL divergence and closer in the MSE of the attention output, yet DimPO achieves better downstream performance. The difference becomes increasingly pronounced once the projection covers more than 50% of the attention layers, where preserving preferences over keys relative to the query, together with head-fidelity on the highest-weighted keys, improves performance on more complex tasks, including SQuAD, common-word extraction, frequent-word extraction, and variable tracking. These results suggest that, under a reduced query and key dimension, preserving the ordering and concentration of task-relevant attention can matter more than minimizing the discrepancy of the full distribution.

Primary area:
foundation or frontier models, including LLMs
Keywords:
preference optimization, attention, listwise ranking, dimensionality reduction, long-context

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_gbMwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DimPO is dimensionality reduction technique that takes inspiration from listwise preference optimization for reducing dimension of query and key spaces. The DimPO objective, which is trained offline, contains the PO term and a top-K cross-entropy term that optimizes for keeping the new probability aligned to the original while pushing the mass to top-K key positions. The evaluations are broad as it contains multiple benchmarks, context lengths, compatibility with different KV reduction methods, and ablations over the layers affected and top-K selection.

Critical strengths
  • The objective of DimPO is well-motivated.
  • Experiments support DimPO’s benefits against its peers on varying context-length along with its compatibility with SnapKV and MagicPig.
  • Ablations over reduced dimension size, layers affected, and top-K selection is good.
Critical weaknesses
  • The notations in section 3.2 are hard to follow as y_i and x are not clearly defined. I understand they are adopted from PO’s literature but be explicit here as the application is non-trivial.
  • Positional encoding is not discussed enough. If RoPE is used then it will depend on the dimension of query and key space so how does DimPO work there? Should the projection be applied before or after applying RoPE? I am assuming that for generalizability of the method to arbitrary PE, DimPO may be applied to raw queries and keys but would like to be clarified.
  • For causal language modeling, are the ranking lists generated using the past keys only? The text is not clear about it.
  • While the evaluations are broad along with ablations, the benefit of DimPO is not yet convincing and well-tested as the accuracy drops with KV memory reduction follows the same trend as its peers without a clear sign of superiority in Fig 5-9. The plot may be misleading me for the worse so please provide numbers for sanity check.
  • The latency chart in Fig 11 is missing the peers. It would be good to see the accuracy-latency tradeoff, if any, of DimPO against its peers.
Questions for authors

No specific question apart from the concerns mentioned in the weaknesses.

1 comment

We thank the reviewer for the positive evaluation and constructive feedback. We appreciate the careful notes on missing technical specifications, notation details, and parts of the text that needed clearer explanations. We have addressed these points directly in the revised manuscript and reuploaded the PDF. Below we also provide detailed point-by-point responses.

1. Notation in Section 3.2

In Section 3.2, xx represents the query vector at a given position, yiy_i denotes the ii-th key vector, and ψi\psi_i is the original attention weight assigned to that key by the frozen model. The preference loss ranks candidate keys relative to the query using projected Q⋅KQ \cdot K scores (si=βlog⁡πθ(yi∣x)s_i = \beta \log \pi_\theta(y_i \mid x)). We have added this explicit notation bridge in the revised section.

2. Positional Encoding (RoPE)

We apply full-dimensional RoPE to QQ and KK first, and only then apply the projection FF to reach dimension d′d'. RoPE is never applied in the reduced space. Because the backbone computes attention scores from post-RoPE representations, DimPO operates in the exact same post-RoPE geometry. Projecting before RoPE would mix frequency pairs and distort positional relationships. This execution order is now stated in Section 3.1.

3. Causal LM and Ranking Lists

Queries at any position tt are ranked against the entire 40964096-token key set, including positions after tt. Because FF is a single position-independent map, it is trained to preserve query--key ranking and preference geometry across full contexts rather than fragmented short prefixes. The causal mask is applied subsequently at inference, identically for both original and projected models. We have explicitly added this clarification to the beginning of Section 3.3 in the revised manuscript.

4. Performance Comparison Across Methods

Because Figures 5–9 in Appendix F split individual harness tasks into narrow subplots, visual scale can obscure absolute gaps. However, Figure 2 in the main body presents the exact 5-task average for LLaMA 8B, showing that DimPO consistently outperforms baseline objectives, particularly pairwise variants.

We have expanded Appendix F in the revised PDF with complete per-task, per-model, and per-layer numerical tables. Across these full metrics, DimPO trails slightly only on TruthfulQA (LLaMA 3B/8B) and occasionally on isolated layers for ARC-Challenge (LLaMA 3B, Qwen 7B) and WinoGrande (LLaMA 8B, Qwen 7B). However, on average, DimPO still achieves the highest overall scores.

Below are the 5-task averages for LLaMA-3.1-8B and Qwen3-4B, alongside ARC-Challenge for LLaMA as a representative task breakdown. Full numerical evaluations for all models and tasks are provided in Appendix F.

LLaMA-3.1-8B — 5-task average (d′=64d'=64)

methodl=0l=8l=16l=20l=24
base66.2————
DimPO(k=64,λ=1)66.265.563.660.257.6
DimPO(k=0,λ=0)66.065.664.360.857.3
KL66.165.362.958.654.0
ORPO66.065.759.748.341.6
CPO66.065.858.748.141.6
SimPO66.065.458.548.041.2
Triplet66.061.841.436.535.3

LLaMA-3.1-8B — ARC-Challenge only (d′=64d'=64)

methodl=0l=8l=16l=20l=24
base55.0————
DimPO(k=64,λ=1)55.054.853.148.646.7
DimPO(k=0,λ=0)55.255.451.749.046.5
KL55.354.451.547.945.6
ORPO55.254.450.538.630.8
CPO55.254.850.238.631.7
SimPO55.253.848.635.730.0
Triplet55.246.925.022.821.2

Qwen3-4B — 5-task average (d′=64d'=64)

methodl=0l=9l=15l=18l=24l=30
base65.7—————
DimPO(k=64,λ=1)65.765.563.762.059.956.8
DimPO(k=0,λ=0)65.765.263.461.858.754.6
KL65.765.162.660.458.355.1
ORPO65.763.751.446.542.038.0
CPO65.764.151.744.541.036.3
SimPO65.764.453.246.141.838.6
Triplet65.760.946.742.539.436.2

5. Latency Trade-off (Figure 11)

All objectives share the exact same architecture, linear projection path, and inference graph. Consequently, runtime latency is identical across methods for a given layer depth ll. The performance difference lies entirely in task accuracy at that shared speedup, which is why Figure 11 visualises DimPO's latency only.

To be completely transparent, below is the full accuracy-latency comparison at layer depth l=16l=16 (~12.5% KV cache memory reduction for LLaMA-3.1-8B). Latency is the mean of 8 \texttt{generate} calls after 1 warmup call, each with batch size 4, an 8192-token prompt and 2048 generated tokens, on a single NVIDIA RTX PRO 6000:

methodlatency(s/batch)↓tok/s↑RULER4kRULER8k
Base(l=0)116.90350.495.0094.16
DimPO(k=64,λ=1)107.97379.490.5287.06
KL107.96379.489.2683.84
SimPO107.86379.85.66—
ORPO107.87379.74.12—
CPO107.82379.95.59—
Triplet107.82379.90.09—

All methods deliver the same ~8.3% speedup at l=16l=16, but DimPO is the only preference-based objective that retains long-context performance.

Reviewer_kwmLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to train low-dimensional linear projections of attention queries and keys in frozen language models. It introduces DimPO, which combines a reference-free listwise preference loss with a top-k cross-entropy term. Projections are trained separately for each layer on BookSum attention traces, then evaluated on four instruction-tuned models using attention-fidelity measures, short-context tasks, and RULER. The paper also reports compatibility with other compression methods and inference latency.

Critical strengths

The controlled comparison of training objectives addresses a useful question: whether matching the full attention distribution preserves downstream behavior. KL training produces lower attention KL and output MSE, yet DimPO performs better on several RULER tasks when more layers are projected. This provides setting-specific evidence that attention-fidelity metrics need not predict task performance under low-dimensional Q/K projection. The component ablations also help separate the effects of listwise ranking and the top-k term.

Critical weaknesses
  1. The novelty of the finding and its advantage over prior projections need clearer clarification. The general observation that teacher–student fidelity need not track downstream performance has appeared in prior distillation work, including Stanton et al., “Does Knowledge Distillation Really Work?” (NeurIPS 2021). DimPO’s narrower contribution is to demonstrate a similar mismatch between attention fidelity and task scores under low-dimensional Q/K projection. The paper also does not compare with Lin et al., “MatryoshkaKV: Adaptive KV Compression via Trainable Orthogonal Projection” (ICLR 2025) at a matched total cache budget, leaving the benefit of its objective over existing projection systems unclear.

  2. The quality–efficiency trade-off is not yet compelling. Values remain at full dimension, so halving key dimension in half the layers reduces total KV-cache storage by about 12.5%. In the reported 8B latency setup, projecting 62.5% of layers cuts runtime from 116.9s to 105.7s; at that same layer fraction, the separate RULER 4k evaluation falls from 95.0 to 52.4. The useful operating range may therefore offer only modest savings.

  3. The layer-selection protocol limits the conclusions. Projections are applied only from the final layer backward. This is a reasonable way to limit error propagation, but the number of layers that can be compressed may depend on that order. The paper does not compare other subsets or an importance-based selection rule.

  4. The empirical scope is narrow. The study uses four 3B–8B instruction-tuned models, trains projections on 4k-token BookSum segments, and evaluates RULER at 4k and 8k. At 50% projected layers, the drop differs considerably across models. It remains unclear whether the reported preference advantage persists for larger models, different training text, or longer contexts.

Questions for authors
  1. How does this finding relate to prior distillation analyses such as Stanton et al. (2021), and what is specific to low-dimensional Q/K projection? Could the authors compare DimPO with MatryoshkaKV at a matched total KV-cache budget?

  2. Can the authors report task quality, cache size, and latency for the same projected-layer configurations, especially where RULER remains near the uncompressed model? This would identify a useful quality–efficiency operating point.

  3. How do random, early-layer, or importance-selected subsets compare with the last-to-first order? This would show whether the conclusions about projection depth depend on the layer-selection rule.

  4. How stable are the results when the training corpus or training context length changes, and when evaluation extends beyond 8k tokens or to larger models? This would test the scope of the empirical finding.

2 comments

We thank the reviewer for their constructive feedback and valuable suggestions regarding method novelty, empirical scope, and scaling analysis. All points have been carefully addressed, and the corresponding results and analyses have been incorporated into the revised manuscript. Below, we address the specific points and concerns with pointers to the respective sections and appendices in the reuploaded PDF.

1. Novelty and Comparison with MatryoshkaKV

Stanton et al. (2021) study agreement between student and teacher predictions. They show that students often disagree with their teachers despite having sufficient capacity, while distillation can still improve generalization. In self-distillation, closer agreement does not necessarily improve generalization, whereas for larger teachers fidelity and generalization tend to improve together.

Our finding is more specific: for offline linear Q/K projection in frozen LLMs, with V kept at full dimension, KL-based training achieves closer attention distributions and outputs, yet DimPO (k=64k=64, λ=1\lambda=1) achieves higher downstream scores. This gap increases with the number of projected layers.

We have added a direct comparison with MatryoshkaKV in Section 4.3. The comparison is performed on RULER 4k with Llama-3.1-8B-Instruct at matched KV-cache budgets.

LayersKV cache savedMatryoshkaKVDimPO
43.125%65.5%94.9%
86.250%65.3%94.2%
129.375%64.8%93.8%
1612.500%63.9%90.5%
2015.625%62.8%52.4%

At 15.6% cache savings, MatryoshkaKV performs better, although both methods are substantially below the uncompressed model (95.0). On a single RTX PRO 6000, this DimPO run takes approximately 1.3 GPU-hours, compared to approximately 77 GPU-hours for MatryoshkaKV.

2. Inference Latency and Quality Trade-off

Just to clarify: our primary goal is introducing and analyzing the DimPO objective (and comparing it against KL and other baselines), rather than building a standalone, production-ready KV-cache compression framework. That said, we agree that verifying its potential practical impact is essential.

We have expanded Appendix H in the revised manuscript to include a full accuracy-latency evaluation table on BOOKSUM (44k prompt with 10241024 generated tokens, and 88k prompt with 20482048 generated tokens, using batch size 4). Below is the summary table for Llama3.1-8B-Instruct (d′=64d'=64) showing how RULER scores, KV cache savings, and end-to-end generation latency scale with the number of projected layers ll:

lllayersKVKKVVRULER(4k)Latency4k(s)RULER(8k)Latency8k(s)
00%0%1.001.0095.0037.7894.16116.90
825%6.2%0.8751.0094.1836.4991.68112.39
1650%12.5%0.7501.0090.5235.1787.06107.97
2062.5%15.6%0.6881.0052.4234.6532.50105.70

As these numbers show, l=16l=16 serves as the optimal operating point: it cuts 88k generation time by ≈8%\approx 8\% while keeping long-context performance right next to the uncompressed base model. Full details, curves, and SnapKV comparisons are now in Appendix H of the revised PDF.

3. Layer-Selection Strategies

To investigate whether our conclusions depend on the layer-selection rule, we have expanded Section 4.5 in the revised manuscript to include an evaluation of different 1616-layer subsets on Llama3.1-8B-Instruct (d′=64d'{=}64). To construct the importance-based subset (importance-16) fairly without using target RULER scores, we evaluate layer sensitivity on four held-out 40964096-token BOOKSUM chapters by measuring the output logit KL divergence between the full model and a single-projected-layer model for each layer individually.

subsetDimPO (k=64,λ=1k{=}64, \lambda{=}1)KL
last-1690.5289.26
importance-1687.2286.20
rand-1648.2928.76
mid-1643.9330.48
early-1614.785.16

As the results show, while last-16 performs best, importance-16 stays very close behind, whereas random, middle, and early subsets drop sharply. This confirms that early layers are significantly more sensitive to perturbation. Furthermore, DimPO consistently outperforms the KL baseline across every subset.

4. Scope & Generalization Concerns

As noted in Section 6 (Limitations), comprehensive evaluations across all dimensions—such as scaling to larger model sizes, longer training contexts, or evaluating on extended sequence lengths—are compute- and resource-intensive due to limited hardware resources. Nevertheless, we fully agree that these comparisons are important to understand how the method behaves and scales under different conditions. To directly test the empirical scope of our findings, we have conducted additional targeted experiments addressing each of these dimensions.

We have updated Section 6 (Limitations) to cross-reference these new evaluations, which are now included in Appendix J.4–J.7 (all evaluated on Llama3.1-8B-Instruct (ℓ=16,d′=64\ell{=}16, d'{=}64) except the Qwen 14B experiment):

Evaluation on Longer Contexts (Appendix J.4)

EvaluationBaseDimPO\mathrm{DimPO} (k=64k{=}64, λ=1\lambda{=}1)KL
RULER 4k95.0390.5289.26
RULER 8k94.1687.0683.84
RULER 16k93.3980.0575.92

Training Corpus Sensitivity (Appendix J.5)

Training corpusRULER 4k (Base)RULER 4k (DimPO\mathrm{DimPO})RULER 4k (KL)RULER 8k (Base)RULER 8k (DimPO\mathrm{DimPO})RULER 8k (KL)
BOOKSUM\texttt{BOOKSUM} 4k95.0390.5289.2693.9487.0683.84
SlimPajama\texttt{SlimPajama} 4k95.0391.4390.2193.9485.9383.73

Evaluation on Larger Model Architecture Qwen2.5-14B-Instruct (Appendix J.6)

ℓ\ellKV RemovedRULER 4k (Base)RULER 4k (DimPO\mathrm{DimPO})RULER 4k (KL)RULER 8k (Base)RULER 8k (DimPO\mathrm{DimPO})RULER 8k (KL)
126.2%94.7094.1693.7593.1591.8390.89
168.3%94.7090.1688.6793.1584.3179.47
189.4%94.7089.3588.0293.1582.1276.73
2412.5%94.7077.9573.9793.1567.1658.55
3015.6%94.7062.3555.3893.1547.2836.49

Training Context Length Sensitivity (Appendix J.7)

Training contextRULER 4k (Base)RULER 4k (DimPO\mathrm{DimPO})RULER 8k (Base)RULER 8k (DimPO\mathrm{DimPO})
BOOKSUM\texttt{BOOKSUM} 4k95.0390.5293.9487.06
BOOKSUM\texttt{BOOKSUM} 8k95.0392.1593.9488.82

These results confirm that DimPO\mathrm{DimPO} remains effective and consistently outperforms KL across longer contexts, different training distributions, larger architectures, and varied training sequence lengths.

Reviewer_Rj9TOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes adapting a preference optimization loss to train performance-preserving channel-wise KV cache compression. By preserving the ranked ordering of attended keys, and the exact attention distribution over only the top-k, effective downscaling projections are produced for stored keys, with minimal and smooth degradation of downstream performance as the degree of intervention is raised. Crucially, the proposed technique is orthogonal to other KV cache compression techniques, and outperforms naïve matching to the attention scores over the full set of keys.

Critical strengths

Idea is sound, straightforward, and sensible. Presentation is direct, clear, and well organized. Results demonstrate consistently high performance against baselines with smooth degradation as compression level and affected layers increase. Analysis looks beyond basic ablations, covering the surprising shortcomings of proxy metrics, compositionality with other compression techniques, and runtime.

Critical weaknesses
  1. One crucial comparison is missing from ablations. Table 6 indicates that the top-k KL divergence loss is useful in raising performance of raw DimPO in at least one setting. Is the converse true? Does the benefit from DimPO come primarily from the ranking loss, the relaxation of KL divergence loss to only the top-k, or the combination? Based on current results, it could be the case that ranking loss is able to broadly recover the performance of KL loss, but offers no benefit above and beyond the relaxed top-k KL loss.

  2. While the partitioned per-layer training makes sense from the perspective of rigorous evaluation, intuitively this would seem to damage performance. If layers are instead allowed to learn to accommodate errors from previous layers, does downstream performance improve? Is there a practical reason not to cover this seemingly more practical setting?

  3. Little to no discussion of the end-to-end KV cache savings. Shrinking keys generates diminishing returns, as the upper bound for compression is, given the constant-size value buffer, 2x. It would then seem more sensible to gently compress a large number of layers, than to aggressively compress a subset. Yet downstream comparisons perform an aggressive 50% reduction on keys, applied to only up to half the layers. Can a comparison be made to the savings-equivalent, yet ostensibly more deployment-friendly, setting of 25% reduction applied to all layers?

Questions for authors

None, see weaknesses

2 comments

We thank the reviewer for their thoughtful and constructive feedback, as well as the valuable insights on how to further strengthen our analysis through additional experiments. In particular, we appreciate catching the missing LheadL_{\mathrm{head}} entry in our main ablation table, which was an oversight on our part and is a crucial component to include.

We have addressed all points raised by the reviewer, incorporated the corresponding new evaluations into the revised manuscript, and re-uploaded the PDF. Below, we detail our specific responses and additions corresponding to each point:

1. Complete Ablation of DimPO Loss Components

To isolate whether DimPO's performance stems from the ranking loss (LlistL_{list}), the top-kk cross-entropy term (LheadL_{head}), or their combination, we have added a row for the isolated LheadL_{head} term directly to the main ablation table in Section 4.5.

The empirical results show that the ranking loss does not merely duplicate the top-kk cross-entropy term; rather, the two terms act complementarily. While LheadL_{head} accounts for most of the base performance, combining both terms into full LDimPOL_{DimPO} achieves the highest overall quality. To illustrate where this synergy is most critical, the table below shows not only the RULER 4k Avg, but also results for Variable Tracking—the RULER subtask where this complementary behavior provides the strongest boost:

ModelObjectiveRULER 4k AvgVariable Tracking
Llama3.1-8B(l=20l{=}20,d′=64d'{=}64)Llist\mathcal{L}_{\mathrm{list}}35.5828.8
Lhead\mathcal{L}_{\mathrm{head}}(k=64k{=}64)50.8727.0
LDimPO\mathcal{L}_{\mathrm{DimPO}}(k=64,λ=1k{=}64,\lambda{=}1)52.4239.6
KL38.9226.7
Qwen3-4B(l=24l{=}24,d′=64d'{=}64)Llist\mathcal{L}_{\mathrm{list}}63.3720.2
Lhead\mathcal{L}_{\mathrm{head}}(k=64k{=}64)63.4732.8
LDimPO\mathcal{L}_{\mathrm{DimPO}}(k=64,λ=1k{=}64,\lambda{=}1)65.9369.6
KL50.2411.2

On both models each term is ahead on some subtasks and behind on others, and the tasks do not line up. For a complete per-subtask breakdown across all RULER evaluation tasks or to see RULER ablation results for different ll, please refer to Appendix J.1 in the revised manuscript.

2. Independent, Sequential, and Joint Training

To address concerns about layer coupling, we have added Appendix J.2 (Independent, sequential, and joint training) comparing independent training (as it is described and done in all experiments in main body of the paper) against sequential (updating layers one at a time from lowest to latest, incorporating prior frozen projections) and joint (updating all projected layers simultaneously) procedures on Llama3.1-8B-Instruct (d′=64d'{=}64) evaluated on RULER 4k:

Protocoll=16l=16l=20l=20
Independent (base)90.5252.42
Sequential87.7738.18
Joint90.6252.81

Independent and joint training yield near-identical performance, while sequential one drops. This confirms that independent training remains a clean and robust protocol for isolating the objective without incurring the overhead of multi-layer optimization.

3. Aggressive and Uniform Key Reduction

To address this concern, we have added Appendix J.3 to the revised manuscript, where we evaluate Llama3.1-8B-Instruct (d=128d=128) under a matched total KV-cache reduction of 12.5%12.5\%. Since values stay at full dimension, cutting key dimensions in half (d′=64d'=64) in the last 1616 of 3232 layers achieves this 12.5%12.5\% savings. We compare this against the savings-equivalent uniform alternative: reducing key dimensions by one-quarter (d′=96d'=96) across all 3232 layers. We also include a controlled baseline applying that same gentler key cut restricted only to the last 1616 layers (6.25%6.25\% total cache reduction):

SettingKeycutLayersKVCacheDimPOKL
Base(ℓ=0\ell=0)--000%0\%95.0095.00
Half keys,last halfd′=64d'{=}64ℓ=16\ell=1612.5%12.5\%90.5289.26
Quarter keys,all layersd′=96d'{=}96ℓ=32\ell=3212.5%12.5\%66.7766.77
Quarter keys,last halfd′=96d'{=}96ℓ=16\ell=166.25%6.25\%94.0994.23

At the same 12.5%12.5\% cache size, applying a mild cut to all layers causes performance to drop sharply to 66.7766.77 for both methods. In contrast, the targeted cut on the last 1616 layers stays much higher at 90.5290.52 (DimPO) and 89.2689.26 (KL). This demonstrates that spreading compression across every layer, including early layers, hurts performance much more than applying a heavier cut only to the later layers.

Reviewer_Rj9TOpenReview ↗

Response to authors

I thank the authors for their response and additional experiments. My concerns are almost all addressed, and the new results are quite interesting (the independent strength but also synergy of the ranking and top-k KL loss components, the high sensitivity of lower layers to compression, and the equivalence of independent and joint layer training). I still agree with fellow reviewers that the performance/overhead tradeoff is not particularly favorable compared to the usual KV-cache compression results, but an orthogonal 8-12% savings in speed and memory is non-trivial, and DimPO consistently scores well compared to baselines targeting the same head dimensionality compression axis. Combined with the interesting empirical findings, I am willing to increase my rating to Accept.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.