acceptodds
Under review as a conference paper at ICLR 2027

Model Compression with Exact Budget Constraints via Riemannian Manifolds

Abstract

Assigning one of K options to each of N groups under a total cost budget is a recurring problem in efficient AI, for instance in mixed-precision quantization, non-uniform pruning, and expert selection. The objective (model loss) depends on all assignments jointly and does not decompose across groups, which means that combinatorial solvers can only optimize proxy objectives. Methods such as evolutionary search evaluate the actual loss but lack gradients, while penalty-based methods enforce the budget only approximately and can require heavy hyperparameter tuning. We show that under softmax relaxation, the budget constraint defines a smooth Riemannian manifold in logit space with unusually clean geometry. Specifically, the normal vector is available in closed form, shifting logits along the cost vector changes expected cost monotonically, and the vector transport reduces to a single inner product. Building on this, we propose Riemannian Constrained Optimization (RCO), which wraps tangent projection, binary-search retraction, and momentum transport around a standard Adam step. Combined with Gumbel straight-through estimation and budget-constrained dynamic programming for discrete feasibility, RCO provides first-order optimization of the actual loss with the expected budget enforced exactly at every iterate, and no constraint-related hyperparameters. The same construction handles several simultaneous budgets, whenever an assignment must respect more than one resource limit, with no added coefficients. RCO exceeds or matches the performance of state-of-the-art methods in both synthetic problems and realistic LLM compression settings, often at considerably lower wall-clock cost.

Primary area:
optimization
Keywords:
Riemannian optimization, discrete assignment, constraint optimization, softmax manifold, geometric retraction

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_z7XoOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2(+1)
Soundness
3(+1)
Read review
Paper summary

The paper proposes a method for budget-constrained model compression, where each model component must be assigned an option, such as a bitwidth or a keep/prune decision, while respecting a global cost limit. The authors show that, under a softmax relaxation, logits with a fixed expected cost form a smooth Riemannian manifold whose geometry enables efficient gradient projection, binary-search retraction, and momentum transport. They incorporate these operations into Adam to develop RCO, combining it with Gumbel straight-through estimation and dynamic programming to obtain discrete assignments. The method optimizes the actual model loss while enforcing the expected budget exactly at every iteration, without requiring constraint-specific penalty tuning. Experiments on multiple-choice knapsack, MoE expert pruning, and mixed-precision quantization show that RCO achieves comparable or better results than existing baselines.

Critical strengths

1. Geometric formulation: The paper characterizes the fixed expected-cost constraint as a manifold, allowing efficient gradient projection through a closed-form normal and a one-dimensional search.

2. Direct optimization of model loss: RCO optimizes the actual model loss using gradients which avoids the reliance on decomposable proxy objectives.

3. Applicability: The method is evaluated on quantization, expert pruning, and mixed-precision MoE compression, and can be integrated with standard first-order optimizers although its effectiveness depends on the quality of the gradient estimator.

Critical weaknesses

1. Presentation. The introduction relies on technical details (the gradient of CC, the variance identity, Riemannian definitions) before the problem and notation are set up, and Section 2 repeats much of this material. Key parts of the method, including the Gumbel/DP/STE pipeline and the LLM setup, appear only later in Section 3.1 and the appendix. Some notation is overloaded: gg denotes both the gradient and the number of Gumbel samples (Algorithm 1), and NN denotes both the number of groups and the matrix of constraint normals.

2. The manifold constraint is not shown to be necessary. The DP decode already makes the final model feasible. In the authors' own penalty comparison (Table 12), a tuned quadratic penalty with DP decoding comes within 1–3% of RCO's KL for a single quantization budget. This comparison is a single run and reports KL only.

3. No ablation of the method's components. Projection, retraction, and momentum transport are not evaluated separately, and no variant is reported that removes the manifold steps and relies only on DP for feasibility.

4. Cost accounting is incomplete. The time to build the GPTQ database and to cache the reference log-probabilities is not reported, nor is the size of the database; the cache alone requires about 159 GB of RAM for 256 samples. Against HIGGS and IMPQ (about 2 h each), RCO's runtime of 62–215 min (Table 2) is comparable rather than clearly lower.

5. Narrow evaluation. The Qwen3-Coder-Next experiment (Table 1) has no external baseline. Dense quantization is evaluated on a single model (Qwen3-8B) with weight-only GPTQ and perplexity only, and FineWeb-Edu is used for both calibration and evaluation.

Questions for authors

1. Necessity of the manifold steps: How does RCO compare with a variant that keeps Gumbel sampling and DP decoding but removes projection, retraction, and transport, and with variants that use each component alone? This would show whether enforcing the budget during optimization matters beyond the DP decode.

2. Seeds and consistency: Which seed is used in Table 2, and why is the same configuration (g=4g=4, T=200T=200, 2.5 bits) reported as 15.60–15.81 FW PPL across Tables 2, 20, 21, and 22? Can the authors report mean and standard deviation over multiple seeds for the main comparisons, including expert pruning?

3. Cost accounting: What are the time and memory costs of building the GPTQ database and caching the reference log-probabilities, and are they included in the reported runtimes? Can the authors provide hardware-matched timings against EvoESAP?

3 comments

We thank the reviewer for their comments. The additional experiments suggested are provided in the General Response (G1-G4) and will be added to the paper (G1 and G3 to the appendix, the seed statistics of G2 to Table 2).

What the paper adds. Under a softmax relaxation, the logits with a fixed expected cost form a smooth manifold with a closed-form normal. Projection is then an inner product, retraction a one-dimensional search, and transport of Adam's momentum one more projection, O(NK)O(NK) per step, so the iterate is on the budget at every step and any number of budgets is held without a coefficient. To our knowledge no prior gradient-based method does this: ProxylessNAS, FBNet, and RC-DARTS enforce expected-cost budgets through penalties or a decaying Lagrangian, which hold the budget only approximately and only after tuning. The new data show the manifold is essential rather than decoration:

  • Same Gumbel-STE and DP, no manifold steps: the iterate drifts to 5.1 bits against a 3.0 target and the decoded model is 3.9x worse in KL (G1).
  • Tuned penalty: close to RCO on one quantization budget (Table 12) but only after a λ\lambda sweep, 1.75x worse on one pruning budget, and with two budgets no shared λ\lambda meets both (Table 11); RCO holds both within 10−410^{-4}.
  • Knapsack benchmark: RCO reaches 1% of the DP optimum, the Lagrangian 37.6%; the 48 per-layer budgets of Qwen3-Coder-Next are held exactly in one run.
  • Scale: on Qwen3-30B-A3B (18,624 layers, 6,336 per-expert units) evolutionary search is out of reach, and only sensitivity-based allocation (HIGGS) was feasible before; RCO searches it directly on the loss and is 17 zero-shot points and 6 FW perplexity points better than HIGGS at 2.25 bits (G4).

W1. Presentation and notation

Response: Section 1 will state the problem and contributions in plain language, without formal statements; Section 2 sets the notation and derives the manifold and the three operations once; Section 3 presents the algorithm with the Gumbel, DP, and STE pipeline; Section 4 opens with the LLM setup (KL objective, GPTQ database, STE). Figure 2 becomes a figure of the budget manifold, one step with projection, retraction, and transport, followed by the Gumbel/DP/STE stages. The Gumbel sample count becomes SS and the normal matrix a calligraphic NN.

W2 / W3 / Q1. Necessity of the manifold steps

Response: The manifold is necessary and we ablate the steps in detail. Both requested ablations are in G1 and summarized above. Removing retraction is the next most harmful (+81% KL, drift to 2.66 bits), retraction alone still leaves +29% that adding projection removes, and transport keeps the optimizer state tangent (Proposition 6).

W4 / Q3. Cost accounting

Response: The GPTQ database (115 GB for Qwen3-8B, 274 GB for Qwen3.8-27B, 2.8 h on 8 RTX A6000 for the 27B) is built once and shared by all methods; it belongs to the quantizer, not to the allocation method. The 2.7 min log-probability cache (159 GB RAM) is the full-vocabulary distribution for 256 x 2048 tokens. Runtimes are search only for every method; we will list GPU types and preprocessing separately. Runtime against HIGGS/IMPQ is comparable; the advantage is quality (Table 2, G3). Hardware-matched EvoESAP timings (46 min vs 11.8 h, one L40S) are in G4.

W5. Narrow evaluation

Response: C4 is never used for calibration and is reported alongside FineWeb-Edu with the same method ordering. Zero-shot accuracy and Qwen3.8-27B (all four methods, both budgets) are in G3: RCO leads EvoPress by 2.6 points at 2.25 bits and 0.5 at 2.5 bits on the 8B; on the 27B RCO has the best perplexity at both budgets and the best accuracy at 2.5 bits. G4 adds Qwen3-30B-A3B with 18,624 layers and 6,336 per-expert units at 2.25 bits: at this granularity only sensitivity-based allocation (HIGGS) was feasible before. RCO reaches 11.03 / 18.09 FW / C4 perplexity and 62.5% zero-shot accuracy against 17.26 / 29.39 and 45.2% for HIGGS (FP16: 9.93 / 15.86, 71.7%). Weight-only GPTQ: the quantizer is an input to RCO, not part of it; activation bitwidths are additional groups with their own options, so weights and activations can receive different bitwidths under one budget, which we consider interesting future work. Qwen3-Coder-Next is the uniform-vs-nonuniform allocation study at 512 experts per layer.

Q2. Seeds and consistency

Response: Table 2 used seed 2. Seed statistics are in G2: RCO (g=32) is below EvoPress at both budgets on both datasets, and g=4 is also below EvoPress at 2.5 bits. For expert pruning (OLMoE-1B-7B, 25%, 3 seeds), RCO reaches 0.587 ±\pm 0.011 average accuracy over 8 benchmarks against 0.566 (REAP uniform) and 0.581 (EvoESAP); the paper's single run is at the upper end of the seed range.

I thank the authors for their thorough response. The ablation studies addressed most of my concerns regarding the necessity of staying on the budget manifold while performing the compression. Although the advantage over the approaches that require a penalty term remains less straightforward to me, I believe that based on the rebuttal discussion and in light of the new results the paper is in an acceptable format and therefore I will raise my initial score to weak acceptance.

We thank the reviewer for raising the score. On the penalty point, briefly:

Many budgets have no slack: e.g. GGUF stores one expert count per model, so every layer must keep exactly the same number of experts; a per-device memory limit is the same kind of constraint. A penalty on its own does not meet them (a layer with 15 or 17 experts instead of 16 cannot be exported at all), so it needs the decode, and the model deployed is then not the one optimized: the coefficients that hold the budget leave 0--5 of 1024 experts committed (Table 13 and below). On the manifold every iterate is on the budget and the decode only rounds it.

With several budgets, no single coefficient holds them and optimizes at the same time. Two budgets on the same experts (Table 11): every λ\lambda misses both, 1.11--3.41x RCO's KL. Sixteen synthetic budgets (Figure 5): per-budget multipliers diverge to violations of order 10 (augmented Lagrangian: 10−210^{-2}), the manifold holds 10−1310^{-13}. We have now also run this on a real model, OLMoE-1B-7B with one budget per layer (exactly 16 of 64 experts pruned in each of the 16 layers, 3 seeds): a shared penalty does not keep the exact per-layer counts during the search (off by up to 1.5 experts per layer at the best λ\lambda, and at the λ\lambda that holds them nothing commits) and reaches 1.04--1.54x RCO's KL; per-layer multipliers diverge. RCO holds all 16 counts within 10−310^{-3} at every step with no coefficient and has the lowest KL.

Furthermore, in the simplest case (one budget), the manifold step is a drop-in replacement for the penalty: three O(NK)O(NK) operations around any first-order optimizer, every iterate exactly feasible, no constraint-related hyperparameter, for any task and gradient estimator (Proposition 6). A penalty, even at its best λ\lambda, is only approximate (0.007--0.017 bits off target on its own, Table 12), its quality can come close (Table 12) or not (Table 13), and the working λ\lambda differs each time (100, none, 0.01). The manifold removes that search and makes feasibility exact.

Reviewer_uzXYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Riemannian Constrained Optimization (RCO) for budget-constrained model compression. By exploiting the geometric structure of the expected-budget constraint, the method combines tangent projection, retraction, and momentum transport with Gumbel-STE and dynamic programming. Experiments on mixed-precision quantization and MoE expert pruning demonstrate competitive compression performance and improved search efficiency.

Critical strengths

1.The paper presents an interesting formulation of budget-constrained model compression using Riemannian optimization, with a mathematically motivated approach to enforcing expected-budget constraints.

2.The experimental results are encouraging, demonstrating competitive compression performance and substantially reduced search time compared with evolutionary methods.

3.The paper provides extensive theoretical discussions and experimental analyses across different compression tasks and constraint settings.

Critical weaknesses

1.Limited justification for Riemannian optimization. Since DP already ensures discrete budget feasibility, the independent contribution of the Riemannian components remains unclear. A direct comparison with standard Adam using the same Gumbel-STE and DP would be helpful.

2.Unclear theoretical justification for Gumbel-STE. The claim that STE bias vanishes as temperature approaches zero may not hold when the globally constrained DP assignment differs from the independent softmax modes.

3.Limited multi-constraint comparisons. The practical experiments mainly compare against quadratic penalties with a shared coefficient. Stronger baselines, such as augmented Lagrangian methods, would make the results more convincing.

Questions for authors
  1. How much performance improvement comes specifically from tangent projection, retraction, and momentum transport? Would standard Adam combined with Gumbel-STE and budget-constrained DP achieve comparable results?

  2. Could the authors further clarify the relationship between the temperature-dependent softmax surrogate and the globally constrained DP assignment, particularly regarding the claimed reduction in STE bias?

2 comments

We thank the reviewer for the positive assessment of the formulation and the efficiency results, and address each point below.


W1 / Q1. Contribution of the Riemannian components

Response: We ran exactly the suggested variant (Adam + Gumbel-STE + budget-constrained DP in the forward pass and at decode, no projection, retraction, transport, or penalty) and each component in isolation (also G1 in the General Response). Qwen3-8B, 3.0 bits, 200 steps, identical hyperparameters; KL of the decoded model (all variants decode to exactly 3.0 bits):

VariantKLContinuous budget during search
RCO (projection + retraction + transport), 3 seeds0.136 ±\pm 0.001exact
Projection + retraction (no transport)0.135exact
Retraction only0.175 (+29%)exact after retraction
Projection + transport (no retraction)0.246 (+81%)drifts to 2.66 bits
Adam + Gumbel-STE + DP only, 2 seeds0.529 ±\pm 0.001 (3.9x)drifts to 5.1 bits

DP feasibility at decode is therefore not sufficient: without the manifold steps the continuous iterate leaves the budget (E[bits] = 5.1 against a 3.0 target), the forward-pass DP then always projects onto the same budget boundary from far away, and the decoded model is 3.9x worse in KL. Staying on the budget yields better solutions because every step optimizes the trade-off within the exact budget that will be deployed, whereas off-budget variants optimize a different problem and decode to a model they never trained under. Projection removes the constraint-normal part of the biased STE gradient, retraction keeps the iterate exactly on the budget, and transport keeps Adam's momentum tangent (Proposition 6); transport's effect on KL is not measurable here because retraction absorbs the small drift, and we keep it for the guarantee. The ablation will be added to the appendix.


W2 / Q2. Gumbel-STE bias and the globally constrained DP assignment

Response: The reviewer is correct. In the forward pass the DP returns the budget-feasible assignment for the perturbed logits; the backward pass uses the per-group softmax of the same logits. As τ\tau →\rightarrow 0 the softmax concentrates on each group's independent argmax, which coincides with the DP assignment only where the budget constraint does not bind. Where it binds, the two differ and a bias remains at any temperature. Our diagnostics confirm this: a bias proxy (StDt/τ2S_t D_t / \tau^2) does not decay monotonically and ends only 1.19x below its peak over 200 steps.

What the method does guarantee is independent of temperature (Proposition 6, appendix): the tangent projection removes the constraint-normal component of any gradient bias before it reaches Adam, so the bias cannot move the iterate off the budget. The tangential bias remains, and we rely on the empirical results for its effect. We will replace "low temperature ... tightening the STE approximation" (Sec. 3.1) with this precise statement.


W3. Multi-constraint baselines

Response: On the multi-constraint MCKP benchmark (q=16q = 16, N=500N = 500, K=32K = 32), RCO satisfies all constraints to ~1e-13, the augmented Lagrangian to ~1e-2, and the plain Lagrangian diverges. On OLMoE with two budgets (pruning + bitwidth), no shared penalty coefficient meets both (1.11-3.41x RCO's KL).

Many simultaneous budgets are a recurring requirement in practice: per-device memory limits, joint sparsity and bitwidth, or equal expert counts per layer. E.g. the last is required for GGUF export, since llama.cpp stores a single expert count per model: on Qwen3-Coder-Next this is 48 equality budgets, and if one layer misses its count the model cannot be loaded. Lagrangian methods meet such constraints only approximately and need one interacting multiplier per budget; RCO holds all 48 exactly at every step without tuning (Appendix E).

Thanks to the authors for their response. My concerns have been addressed. Although I am not an expert in this field, based on my understanding of the paper, I lean toward acceptance.

Reviewer_efJnOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper has studied the optimization of model compression configurations under resource budget constraints, and then proposed Riemannian Constrained Optimization (RCO). The proposed method uses a softmax parameterization to construct a budget manifold and maintains the continuous expected budget through tangent projection, retraction, and vector transport by projection of Adam’s first moment. The proposed method combines tangent projection, retraction, and momentum transport with Gumbel-STE and dynamic programming, which exploits the geometric structure of the expected-budget constraint. The proposed method is evaluated by experiments on mixed-precision quantization and MoE expert pruning, which demonstrate competitive compression performance and improved search efficiency.

Critical strengths

S1. Rather than simply applying an existing optimizer to model compression, the paper exploits the specific geometry of budget constraints to implement computationally inexpensive tangent projection, retraction, and momentum transport. S2. The experiments provide substantial support for the core claims in different parts. Section 4.1 compares different constraint-handling methods under controlled conditions. Appendix E further evaluates RCO under both single-budget and multiple-budget settings. Table 11 shows that although the final decoded discrete models from all compared methods satisfy both budgets, RCO maintains the budgets more accurately during continuous optimization and achieves a lower final KL divergence. S3. In the comparison, the proposed RCO requires substantially less search time while maintaining competitive compression quality, which can support the paper’s main claim regarding efficient joint optimization.

Critical weaknesses

W1. Some quantization gains are small, and their stability requires more direct evidence from repeated runs. In Table 2, it shows that different RCO search configurations produce somewhat different results. For example, RCO with g=4 and T=200 achieves 15.78, slightly worse than EvoPress. These differences across configurations do not themselves demonstrate random variability, but the smaller margins still warrant repeated comparisons under the corresponding configurations. W2. Appendix F.7 has reported the experimental results with different random seeds, showing that the authors have considered search variability. However, these experiments do not directly replace repeated comparisons of the main methods under their corresponding configurations in the main table.

Questions for authors

Q1. Could the authors provide results across multiple random seeds for the key RCO–EvoPress comparisons in experiments to establish whether the smaller quality improvements are stable? Q2. For non-integer costs in quantization, Appendix B.1 explains that the DP uses an integer cost grid. Could the authors clarify a bound on the true budget deviation introduced by this cost mapping, and how grid resolution affects budget deviation and computational overhead?

4 comments

We thank the reviewer for the careful reading, including Tables 11 and 13, and address both points below.

W1 / Q1. Multi-seed stability of the Table 2 RCO-EvoPress comparisons

Response: We reran the main-table configurations under identical settings: RCO with 3 seeds, EvoPress (100 generations) with 2 new seeds, one code version and one evaluation pipeline. Perplexity, mean ±\pm std (also G2 in the General Response; zero-shot accuracy and Qwen3.8-27B results in G3):

Method2.5 bits FW2.5 bits C42.25 bits FW2.25 bits C4
RCO (g=32, T=50)15.16 ±\pm 0.1223.75 ±\pm 0.3721.07 ±\pm 0.0333.44 ±\pm 0.10
RCO (g=4, T=200)15.33 ±\pm 0.0723.78 ±\pm 0.1621.33 ±\pm 0.3933.93 ±\pm 0.75
EvoPress (100 gen)15.45 ±\pm 0.1124.25 ±\pm 0.3321.55 ±\pm 0.1833.49 ±\pm 0.07

These statistics will replace the single-seed numbers in Table 2.

RCO (g=32, T=50) is better than EvoPress in every column, and the margin exceeds the seed spread on FineWeb-Edu at both budgets and on C4 at 2.5 bits; at 2.25 bits on C4 the two are within noise. Relative to Table 2 both methods move by a similar amount under the rerun (RCO 15.43 to 15.16, EvoPress 15.64 to 15.45), so the shift comes from the code version and pipeline, not from the comparison. We read the small margins as both searches reaching the same flat optimum region: different EvoPress seeds differ by 13-17% in per-layer bitwidths at nearly identical perplexity, so many configurations are equally good, and two very different searches agreeing on that level is the evidence that both are near it.

The key difference is cost and scaling of the methods, and it follows from how the two methods search (G4). EvoPress is a black-box search: each candidate is scored by forward passes that return one scalar, and mutations are random, so they carry no information about which units matter. Its convergence guarantee (Sieberling et al., 2025, Thm. 3.1, for linear fitness) is paid for in evaluations: the O(k(n−k))O(k(n-k)) bound grows with the number of decision units, and each generation changes at most three of them. RCO is gradient-based: one forward and backward pass gives the gradient for all N⋅KN \cdot K logits at once, so every step moves every unit, and neither the cost per step nor the number of steps grows with NN. This is what keeps the search feasible when NN reaches the thousands, as in per-expert quantization of MoEs: on OLMoE-1B-7B (1,024 experts and 16 attention groups, 5 options each, Table 11) RCO runs the same 200 steps as for the 252 layers of Qwen3-8B. For EvoPress the number of generations grows with NN: each generation changes only a few layers, so 100 generations revise at most a few hundred decisions, which covers 252 layers but not thousands.

Q2. DP integer cost grid and budget deviation

Response: Each group's cost is rounded to the nearest point of a grid with rr points per average bit, so the per-group error is at most 1/(2r)1/(2r) and the decoded budget deviates from the target by at most G/(2r)G/(2r) for GG groups. Rounding errors of different groups largely cancel, and the observed deviation is 4-6x below this bound. The grid only enters the decode of a discrete assignment; the budget of the continuous iterate is held exactly by the retraction. For Qwen3-8B the deployed models deviate by +0.0018 bits (2.5-bit target) and +0.0048 bits (2.25), under 0.2% of the budget.

Resolution sweep with the released DP code on synthetic instances with the Qwen3-8B layer sizes (252 layers, bitwidths 2-8, random logits; 5 instances at each of 2.25, 2.5, 3.5, 4.0 bits). Time per DP call, pure Python, one CPU core:

rmax |dev| (bits)bound G/(2r)G/(2r)DP time
2500.0800.500.3 s
5000.0670.250.6 s
25000.0110.0503 s
100000.00300.01314 s
200000.00120.006317 s

The bound holds in every run, and the cost grows at most linearly in rr (O(NKB′)O(NKB') with B′=rbˉB' = r\bar{b}). The grid is not the lever on runtime, the forward pass is: inside the search the DP is about 14% of a step at r=500r = 500 (Appendix, solver-cost table), the rest is the forward and backward pass, and the decode runs once, seconds against 1-2 h of search. For models with many more groups, where the bound grows with GG, rr is raised at a cost that stays small next to the forward passes; for expert pruning the costs are integers (0/1) and the grid is exact. If a budget must never be exceeded, the target is lowered by G/(2r)G/(2r) and the decoded model is guaranteed to be at or below it.

Please engage with the authors during the discussion period.

Thanks to the authors for response. The DP deviation bound G/(2r) and the accompanying sweep cleanly answer Q2. On W1 the response is some thinner, and three of four margins fall within the reported seed spread, with the EvoPress re-runs additionally confounded by code-version and pipeline changes. My concerns about robustness have been alleviated in principle, but have not been proven in the experiments that support the main points of this paper. I will keep my rating unchanged

We thank the reviewer for the reply. Two clarifications on W1, since we may have stated the result unclearly.

Margins versus seed spread. For the main configuration (g=32, T=50) the margin over EvoPress, in units of the pooled seed standard deviation, is 1.8 (2.5 bits, FW), 1.0 (2.5 bits, C4), 2.6 (2.25 bits, FW), and 0.4 (2.25 bits, C4). One of the four is within noise, not three. The reruns are not confounded: RCO and EvoPress were both rerun under one code version and one evaluation pipeline, so the comparison between them is clean; only the comparison to the old Table 2 numbers shifts, and by the same amount for both methods.

What the comparison is meant to show. Qwen3-8B with 252 decision units is a problem EvoPress solves well given 11-14 h: different EvoPress seeds differ by 13-17% in per-layer bitwidths at nearly identical perplexity, so the optimum region is flat and no method will beat it by much. The paper's claim on this instance is that RCO matches or exceeds EvoPress at 6-10x lower wall-clock on the same GPU (62 min vs 14 h, Table 2), and the seed statistics confirm it. The main point is what happens when the number of decision units grows. Expert pruning on OLMoE-1B-7B, both released codes end to end on one L40S: RCO 46 min, EvoESAP 11.8 h (G4).

To the best of our knowledge, non-uniform allocation on the actual loss has so far been done with zeroth-order search (e.g. EvoPress, EvoESAP); everything else optimizes a per-layer sensitivity surrogate instead of the loss (e.g. HIGGS, IMPQ). RCO is the first method to search the loss directly with gradients, and this is what makes problems with thousands of decision units feasible. We added per-expert quantization of Qwen3-30B-A3B with 18,624 layers and 6,336 units (G4) to show the infeasibility of zeroth-order search at this size: each EvoPress generation keeps one survivor that differs from its parent in 1-3 layers moved up plus the layers moved down to restore the budget, about 3-6 layer changes per generation, so 100 generations touch 300-600 layers, every layer of Qwen3-8B once or twice but 1.6-3.2% of the 18,624 layers of the 30B (150 generations: under 5%). RCO searched it directly and improves on the only feasible alternative, sensitivity-based HIGGS, by 17 zero-shot points and 6 FW perplexity points. The Qwen3-30B-A3B comparison will be added to the paper. The robustness the reviewer asks for is established where the margins are small (Qwen3-8B), and where the problem is large the evolutionary baseline does not work.

Comments on the paper

1 comment

General Response

We thank all reviewers. Points raised by more than one reviewer are collected here as G1-G4 and referenced from the individual responses.

G1. Component ablation (z7Xo W2/W3/Q1, uzXY W1/Q1). Reviewers z7Xo and uzXY ask whether the budget manifold is necessary when the DP already makes the decoded model feasible. The gradient of the loss is unconstrained: it asks for a larger budget in every group, so without the manifold the continuous iterate drifts away from the budget at every step, and the budget constraint is what makes the search an allocation problem at all. Optimizing on the manifold removes the budget-changing component from every gradient and from Adam's state. What remains is the relative sensitivity, the loss reduction one unit of budget buys in a given group compared with spending it in any other group at the same total cost. Every iterate is therefore a feasible reallocation, and the DP at decode only rounds a point that already lies on the budget.

Qwen3-8B, 3.0 bits, 200 steps, identical hyperparameters; KL of the decoded model (all variants decode to exactly 3.0 bits):

VariantKLContinuous budget during search
RCO (projection + retraction + transport), 3 seeds0.136 ±\pm 0.001exact
Projection + retraction (no transport)0.135exact
Retraction only0.175 (+29%)exact after retraction
Projection + transport (no retraction)0.246 (+81%)drifts to 2.66 bits
Adam + Gumbel-STE + DP only (no manifold, no penalty), 2 seeds0.529 ±\pm 0.001 (3.9x)drifts to 5.1 bits

DP feasibility at decode is not sufficient. With the same Gumbel-STE and DP but without the budget manifold the continuous iterate leaves the budget (5.1 bits against a 3.0 target), the forward-pass DP then always projects onto the budget boundary from far away, and the decoded model is 3.9x worse in KL. Each component has a distinct role:

  • Tangent projection removes the constraint-normal part of the biased STE gradient, so the optimizer never sees it.
  • Retraction puts the iterate back exactly on the budget after each Adam step.
  • Momentum transport keeps Adam's momentum tangent after each retraction, so bias cannot accumulate along the normal in the optimizer state (Proposition 6).

G2. Seeds for the Table 2 comparison (efJn W1/Q1, z7Xo Q2). Same settings as Table 2, FW / C4 perplexity, mean ±\pm std:

Method2.5 bits2.25 bits
RCO (g=32, T=50), 3 seeds15.16 ±\pm 0.12 / 23.75 ±\pm 0.3721.07 ±\pm 0.03 / 33.44 ±\pm 0.10
RCO (g=4, T=200), 3 seeds15.33 ±\pm 0.07 / 23.78 ±\pm 0.1621.33 ±\pm 0.39 / 33.93 ±\pm 0.75
EvoPress (100 gen), 2 seeds15.45 ±\pm 0.11 / 24.25 ±\pm 0.3321.55 ±\pm 0.18 / 33.49 ±\pm 0.07

Expert pruning (OLMoE-1B-7B, 25%, 3 seeds): 0.587 ±\pm 0.011 average accuracy over 8 benchmarks against 0.566 (REAP uniform) and 0.581 (EvoESAP).

G3. Zero-shot accuracy and a second dense model (z7Xo W5). Mean zero-shot accuracy over six lm-eval tasks (PIQA, HellaSwag, ARC-Easy, ARC-Challenge, Winogrande, LAMBADA-OpenAI; %) and perplexity (FW / C4) for the Table 2 configurations of Qwen3-8B (FP16 acc. 70.4) and Qwen3.8-27B (FP16 acc. 74.1, PPL 8.39 / 11.84), one seed:

Method8B acc. 2.258B acc. 2.527B acc. 2.527B PPL 2.2527B PPL 2.5
RCO (g=32, T=50)47.254.471.210.98 / 16.009.71 / 14.00
EvoPress44.653.970.811.32 / 16.519.90 / 14.31
IMPQ41.852.970.011.25 / 16.429.95 / 14.30
HIGGS35.146.166.013.15 / 19.6211.79 / 17.28

G4. Cost and scaling (z7Xo W4/Q3, efJn W1). Hardware: NVIDIA L40S and RTX A6000. Reported runtimes are search only for every method; preprocessing (GPTQ database, log-probability cache) is built once and shared by all methods. Hardware-matched EvoESAP (OLMoE-1B-7B, 25%), both released codes, end to end on one L40S: RCO (300 steps) 46 min including loading and caching, EvoESAP 11.8 h (5.7 h REAP calibration, 6.1 h search). EvoPress and EvoESAP are black-box: the number of evaluations grows with the number of decision units NN. RCO is gradient-based: one backward pass moves every unit, and neither the cost per step nor the number of steps (50-300 for 252 layers and 1024 experts alike) grows with NN.

Scaling test: Qwen3-30B-A3B, 18,624 layers and 6,336 per-expert decision units, target 2.25 bits. Evolutionary search is out of reach at this scale. To our knowledge, allocation at this granularity was only possible with sensitivity-based methods such as HIGGS; RCO is the first to search it directly on the loss, and the table shows what the loss signal buys (per-expert groups, batch 2; zero-shot as in G3):

Qwen3-30B-A3B, 2.25 bitsFWC4Zero-shot
FP169.9315.8671.7
HIGGS17.2629.3945.2
RCO (g=4, T=400)11.0318.0962.5

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.