- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2(+1)
- Soundness
- 3(+1)
Read review
Paper summary
The paper proposes a method for budget-constrained model compression, where each model component must be assigned an option, such as a bitwidth or a keep/prune decision, while respecting a global cost limit. The authors show that, under a softmax relaxation, logits with a fixed expected cost form a smooth Riemannian manifold whose geometry enables efficient gradient projection, binary-search retraction, and momentum transport. They incorporate these operations into Adam to develop RCO, combining it with Gumbel straight-through estimation and dynamic programming to obtain discrete assignments. The method optimizes the actual model loss while enforcing the expected budget exactly at every iteration, without requiring constraint-specific penalty tuning. Experiments on multiple-choice knapsack, MoE expert pruning, and mixed-precision quantization show that RCO achieves comparable or better results than existing baselines.
Critical strengths
1. Geometric formulation: The paper characterizes the fixed expected-cost constraint as a manifold, allowing efficient gradient projection through a closed-form normal and a one-dimensional search.
2. Direct optimization of model loss: RCO optimizes the actual model loss using gradients which avoids the reliance on decomposable proxy objectives.
3. Applicability: The method is evaluated on quantization, expert pruning, and mixed-precision MoE compression, and can be integrated with standard first-order optimizers although its effectiveness depends on the quality of the gradient estimator.
Critical weaknesses
1. Presentation. The introduction relies on technical details (the gradient of , the variance identity, Riemannian definitions) before the problem and notation are set up, and Section 2 repeats much of this material. Key parts of the method, including the Gumbel/DP/STE pipeline and the LLM setup, appear only later in Section 3.1 and the appendix. Some notation is overloaded: denotes both the gradient and the number of Gumbel samples (Algorithm 1), and denotes both the number of groups and the matrix of constraint normals.
2. The manifold constraint is not shown to be necessary. The DP decode already makes the final model feasible. In the authors' own penalty comparison (Table 12), a tuned quadratic penalty with DP decoding comes within 1–3% of RCO's KL for a single quantization budget. This comparison is a single run and reports KL only.
3. No ablation of the method's components. Projection, retraction, and momentum transport are not evaluated separately, and no variant is reported that removes the manifold steps and relies only on DP for feasibility.
4. Cost accounting is incomplete. The time to build the GPTQ database and to cache the reference log-probabilities is not reported, nor is the size of the database; the cache alone requires about 159 GB of RAM for 256 samples. Against HIGGS and IMPQ (about 2 h each), RCO's runtime of 62–215 min (Table 2) is comparable rather than clearly lower.
5. Narrow evaluation. The Qwen3-Coder-Next experiment (Table 1) has no external baseline. Dense quantization is evaluated on a single model (Qwen3-8B) with weight-only GPTQ and perplexity only, and FineWeb-Edu is used for both calibration and evaluation.
Questions for authors
1. Necessity of the manifold steps: How does RCO compare with a variant that keeps Gumbel sampling and DP decoding but removes projection, retraction, and transport, and with variants that use each component alone? This would show whether enforcing the budget during optimization matters beyond the DP decode.
2. Seeds and consistency: Which seed is used in Table 2, and why is the same configuration (, , 2.5 bits) reported as 15.60–15.81 FW PPL across Tables 2, 20, 21, and 22? Can the authors report mean and standard deviation over multiple seeds for the main comparisons, including expert pruning?
3. Cost accounting: What are the time and memory costs of building the GPTQ database and caching the reference log-probabilities, and are they included in the reported runtimes? Can the authors provide hardware-matched timings against EvoESAP?
3 comments
I thank the authors for their thorough response. The ablation studies addressed most of my concerns regarding the necessity of staying on the budget manifold while performing the compression. Although the advantage over the approaches that require a penalty term remains less straightforward to me, I believe that based on the rebuttal discussion and in light of the new results the paper is in an acceptable format and therefore I will raise my initial score to weak acceptance.
We thank the reviewer for raising the score. On the penalty point, briefly:
Many budgets have no slack: e.g. GGUF stores one expert count per model, so every layer must keep exactly the same number of experts; a per-device memory limit is the same kind of constraint. A penalty on its own does not meet them (a layer with 15 or 17 experts instead of 16 cannot be exported at all), so it needs the decode, and the model deployed is then not the one optimized: the coefficients that hold the budget leave 0--5 of 1024 experts committed (Table 13 and below). On the manifold every iterate is on the budget and the decode only rounds it.
With several budgets, no single coefficient holds them and optimizes at the same time. Two budgets on the same experts (Table 11): every misses both, 1.11--3.41x RCO's KL. Sixteen synthetic budgets (Figure 5): per-budget multipliers diverge to violations of order 10 (augmented Lagrangian: ), the manifold holds . We have now also run this on a real model, OLMoE-1B-7B with one budget per layer (exactly 16 of 64 experts pruned in each of the 16 layers, 3 seeds): a shared penalty does not keep the exact per-layer counts during the search (off by up to 1.5 experts per layer at the best , and at the that holds them nothing commits) and reaches 1.04--1.54x RCO's KL; per-layer multipliers diverge. RCO holds all 16 counts within at every step with no coefficient and has the lowest KL.
Furthermore, in the simplest case (one budget), the manifold step is a drop-in replacement for the penalty: three operations around any first-order optimizer, every iterate exactly feasible, no constraint-related hyperparameter, for any task and gradient estimator (Proposition 6). A penalty, even at its best , is only approximate (0.007--0.017 bits off target on its own, Table 12), its quality can come close (Table 12) or not (Table 13), and the working differs each time (100, none, 0.01). The manifold removes that search and makes feasibility exact.
We thank the reviewer for their comments. The additional experiments suggested are provided in the General Response (G1-G4) and will be added to the paper (G1 and G3 to the appendix, the seed statistics of G2 to Table 2).
What the paper adds. Under a softmax relaxation, the logits with a fixed expected cost form a smooth manifold with a closed-form normal. Projection is then an inner product, retraction a one-dimensional search, and transport of Adam's momentum one more projection, O(NK) per step, so the iterate is on the budget at every step and any number of budgets is held without a coefficient. To our knowledge no prior gradient-based method does this: ProxylessNAS, FBNet, and RC-DARTS enforce expected-cost budgets through penalties or a decaying Lagrangian, which hold the budget only approximately and only after tuning. The new data show the manifold is essential rather than decoration:
W1. Presentation and notation
Response: Section 1 will state the problem and contributions in plain language, without formal statements; Section 2 sets the notation and derives the manifold and the three operations once; Section 3 presents the algorithm with the Gumbel, DP, and STE pipeline; Section 4 opens with the LLM setup (KL objective, GPTQ database, STE). Figure 2 becomes a figure of the budget manifold, one step with projection, retraction, and transport, followed by the Gumbel/DP/STE stages. The Gumbel sample count becomes S and the normal matrix a calligraphic N.
W2 / W3 / Q1. Necessity of the manifold steps
Response: The manifold is necessary and we ablate the steps in detail. Both requested ablations are in G1 and summarized above. Removing retraction is the next most harmful (+81% KL, drift to 2.66 bits), retraction alone still leaves +29% that adding projection removes, and transport keeps the optimizer state tangent (Proposition 6).
W4 / Q3. Cost accounting
Response: The GPTQ database (115 GB for Qwen3-8B, 274 GB for Qwen3.8-27B, 2.8 h on 8 RTX A6000 for the 27B) is built once and shared by all methods; it belongs to the quantizer, not to the allocation method. The 2.7 min log-probability cache (159 GB RAM) is the full-vocabulary distribution for 256 x 2048 tokens. Runtimes are search only for every method; we will list GPU types and preprocessing separately. Runtime against HIGGS/IMPQ is comparable; the advantage is quality (Table 2, G3). Hardware-matched EvoESAP timings (46 min vs 11.8 h, one L40S) are in G4.
W5. Narrow evaluation
Response: C4 is never used for calibration and is reported alongside FineWeb-Edu with the same method ordering. Zero-shot accuracy and Qwen3.8-27B (all four methods, both budgets) are in G3: RCO leads EvoPress by 2.6 points at 2.25 bits and 0.5 at 2.5 bits on the 8B; on the 27B RCO has the best perplexity at both budgets and the best accuracy at 2.5 bits. G4 adds Qwen3-30B-A3B with 18,624 layers and 6,336 per-expert units at 2.25 bits: at this granularity only sensitivity-based allocation (HIGGS) was feasible before. RCO reaches 11.03 / 18.09 FW / C4 perplexity and 62.5% zero-shot accuracy against 17.26 / 29.39 and 45.2% for HIGGS (FP16: 9.93 / 15.86, 71.7%). Weight-only GPTQ: the quantizer is an input to RCO, not part of it; activation bitwidths are additional groups with their own options, so weights and activations can receive different bitwidths under one budget, which we consider interesting future work. Qwen3-Coder-Next is the uniform-vs-nonuniform allocation study at 512 experts per layer.
Q2. Seeds and consistency
Response: Table 2 used seed 2. Seed statistics are in G2: RCO (g=32) is below EvoPress at both budgets on both datasets, and g=4 is also below EvoPress at 2.5 bits. For expert pruning (OLMoE-1B-7B, 25%, 3 seeds), RCO reaches 0.587 ± 0.011 average accuracy over 8 benchmarks against 0.566 (REAP uniform) and 0.581 (EvoESAP); the paper's single run is at the upper end of the seed range.