acceptodds
Under review as a conference paper at ICLR 2027

Not How Many, But Which: Parameter Placement in Low-Rank Adaptation

Abstract

LoRA introduces low-rank adapter parameters for fine-tuning. We study whether, given a fixed budget of trainable entries in the matrix of a LoRA adapter ( frozen), it matters which entries are trained. We call this the *parameter placement problem*: which entries should be trained? Under supervised fine-tuning, random and informed subsets perform comparably in most settings. Under GRPO on base models, random placement fails to improve over the base model, whereas, on mathematical reasoning and on text-to-SQL, gradient-informed placement recovers standard LoRA accuracy. The gradient structure explains this behavior: SFT gradients are low-rank and directionally stable, so any subset accumulates coherent updates; GRPO gradients are higher-rank and less aligned across steps, so only elements with consistently signed gradients retain the learning signal. Our scoring procedure identifies these parameters in under 10 seconds at less than 0.5% of training cost. The selected parameters are concentrated in the attention value projection matrices across model families and scales (1.5B - 7B).

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
lora, parameter efficient fine-tuning, grpo, sparse fine-tuning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_MfEJOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper studies the parameter placement problem in LoRA. Given a fixed budget of kk trainable entries in the BB matrix of a LoRA adapter, with AA frozen and B=0B=0 at initialization, the authors ask whether the choice of entries affects downstream performance. Entries are scored at initialization from 50 forward-backward passes of the supervised cross-entropy loss, using either the absolute mean gradient S^ij=∣1N∑ngn,ij∣\hat S_{ij}=|\frac{1}{N}\sum_n g_{n,ij}| or the diagonal Fisher F^ij=1N∑ngn,ij2\hat F_{ij}=\frac{1}{N}\sum_n g_{n,ij}^2, and the global top-kk entries are trained with α=32r\alpha=32r. The authors report that under SFT random and informed subsets perform comparably (Table 2), whereas under GRPO on Qwen2.5 base models (1.5B, 3B, 7B) random subsets remain close to the base model and S^\hat S or F^\hat F circuits of 10K to 50K entries match full LoRA on GSM8K and MATH-500 (Table 3). A similar ordering is reported for text-to-SQL (Table 36). The proposed explanation rests on gradient structure measured at initialization. SFT gradients are reported to be low-rank and aligned across consecutive steps, whereas GRPO gradients are higher-rank and nearly orthogonal, so that only coordinates with consistently signed gradients accumulate displacement (Table 1). A synthetic two-layer MLP experiment is presented as reproducing this distinction (Fig. 1). The remaining analyses characterize the selected circuits. Discovery is stable and takes under 10 seconds (Tables 6 to 8), random placement remains at base accuracy for α\alpha between 32 and 2048 (Table 13), circuits overrepresent the V projection and concentrate in early layers (Table 4), removing high-scoring entries from a trained full-LoRA adapter degrades accuracy faster than random removal (Fig. 3), and circuits induce larger logit-lens distribution shifts than random subsets at comparable weight norm (Fig. 4).

Critical strengths
  1. The central empirical observation under GRPO is clear and is controlled for the most obvious alternative explanation. On Qwen2.5 base models, random sparse placement remains at or near base accuracy (on 1.5B GSM8K, 9.5 for the base model, 11.6±3.111.6\pm3.1 for random, 65.2±1.765.2\pm1.7 for S^\hat S and 63.7±1.863.7\pm1.8 for full LoRA, Table 3), while the informed circuits recover full-LoRA accuracy with well under 1% of its trainable parameters. A natural concern is that a random mask receives too small an effective update. Table 13 addresses this directly, since at k=10k=10K random placement stays between 9.1% and 9.9% for every α\alpha from 32 to 2048, while the circuits improve monotonically and saturate near α=32r\alpha=32r. The same ordering is observed on text-to-SQL with an execution reward across four Qwen2.5-7B variants (Table 36). This result is the main reason I do not recommend a clear rejection. It is of practical relevance, as it indicates that under GRPO on base models the trainable entries of a sparse adapter cannot be chosen arbitrarily, and that a short gradient pass at initialization suffices to identify a working subset. It also qualifies the conclusion of Donoway et al. (2025) that randomly chosen subsets are sufficient.

  2. The appendix reports several results that are unfavorable to the main narrative, which increases confidence in the reported measurements. The authors show that random placement fails under SFT on GSM8K (Table 31), that circuits scored with cross-entropy and with GRPO share only 11 to 13% of their entries (Table 20), and that normalizing scores within each tensor renders the knockout curve indistinguishable from random (§D.8). These results are reported transparently. As the weaknesses below indicate, they are also central to evaluating the paper's claims, and much of the evidence needed to revise the interpretation is already present in the submission.

Critical weaknesses
  1. The main explanatory claim, that placement matters under GRPO but not under SFT because GRPO gradients are noisy and cancel across steps, is not supported by the paper's controlled evidence, and the proposed mechanism is not tested by intervention. The SFT evidence in Table 2 is obtained on Alpaca and evaluated with log-likelihood multiple-choice benchmarks, on which full LoRA itself yields little improvement over the base model (65.6 to 66.7 for Qwen2.5-3B and 70.9 to 71.5 for 7B). With roughly one point of headroom, all placements are expected to appear comparable. The GRPO evidence, in contrast, is obtained on generative mathematical reasoning with exact-match grading. The training regime is therefore confounded with the task and the evaluation metric. The only same-task comparison in the paper, Table 31, does not support the claimed distinction. Under SFT on GSM8K with Qwen2.5-1.5B, random placement reaches 13.2% against a base accuracy of 9.5%, while F^\hat F, S^\hat S and full LoRA reach 62.2%, 64.3% and 61.4%. Random placement thus fails under SFT in the same way as under GRPO (11.6%, Table 3). Qwen2.5-3B exhibits the same pattern (random 62.9% against a base of 62.1%), and random placement falls below the base model on IFEval after SFT (Table 33). This failure occurs in precisely the setting for which Table 17 reports favorable SFT gradient structure (effective rank 2.3, consecutive cosine similarity 0.87, net-to-step norm ratio 0.93), so the proposed mechanism predicts the opposite outcome. The converse case is also inconsistent with the mechanism. For Qwen2.5-7B on GSM8K the GRPO gradient is comparatively well aligned (cosine 0.41, net-to-step ratio 0.78, Table 17), yet random placement remains near the base model (80.9% against 79.3%, Table 3). The SFT comparison is further complicated by an inconsistency in the reported scaling factor. Section 4 states that α=32r\alpha=32r is used for every sparse run, random placement included, whereas §E.4 states that the random baselines in Table 2 use α=32\alpha=32. The latter would give random placement a 32-fold smaller effective step, which Table 13 shows is sufficient to keep even the informed circuits at base accuracy under GRPO.

More fundamentally, gradient noise is proposed as the cause, but it is never varied experimentally. All gradient statistics are computed at initialization, where B=0B=0 and the parameters do not change between the compared steps. Table 1 therefore measures agreement between minibatch gradients at a single point rather than cancellation along a training trajectory. If the batching in Table 27 follows the TRL convention, each GRPO update contains approximately 2 prompts with 4 completions each, and at 9.5% base accuracy most groups receive zero advantage. Near-orthogonal consecutive gradients may thus be a consequence of the small effective batch rather than a property of the RL objective. The experiment that would test the mechanism directly is absent. Such an experiment would increase the number of prompts or generations per GRPO step and examine whether random placement recovers as gradient noise decreases, and would conversely reduce the batch size or inject gradient noise under SFT and examine whether random placement begins to fail. The synthetic experiment does not substitute for this test. Between its two settings it simultaneously changes the target (a dense rank-2 residual versus a B∗B^* with 5% large entries), the optimizer (Adam versus SGD), the batch size (128 versus 4) and the noise level (none versus σ=0.005\sigma=0.005) (§C.6). Because the RL-like target is sparse in the coordinate basis of BB by construction, the advantage of coordinate selection in that setting is built into the design. It also models the GRPO signal as concentrated, whereas Section 3.3 describes the GRPO gradient as spread over more directions. In addition, Fig. 1 does not agree with the accompanying text. In Fig. 1(a), informed placement achieves lower relative MSE than random placement by approximately 0.15 to 0.25 at every budget (approximately 0.57 versus 0.87 at 10%), contrary to the statement that the two are "indistinguishable" (line 208). In Fig. 1(b), random placement at 50% reaches a relative MSE of approximately 0.38, contrary to the caption's statement that it "barely improves even at 50%". Since the title, abstract, Section 3.3 and conclusion all rely on this regime distinction, this is the weakness that most strongly determines my rating.

  1. Whereas the first weakness concerns when placement matters, the second concerns what the selection procedure identifies. The evidence in the paper is consistent with explanations at a coarser granularity than the individual entries emphasized in the title, and several supporting analyses are circular in a way that obscures this possibility. The knockout experiment ranks the entries of a trained full-LoRA adapter by their initialization scores and interprets faster degradation as causal evidence that the scored parameters mediate the adapter's effect (line 424). However, §D.8 shows that z-scoring the scores within each tensor, which preserves the within-module ranking and removes only the cross-module scale, makes the knockout curve indistinguishable from random removal. The knockout curve therefore indicates that some modules matter more than others, which the global top-kk selection already encodes. At fractions comparable to the circuit size (0.1% to 1%) the accuracy drop is small, and the gap emerges only between 20% and 75% removal. The interpretation of selected modules as attention readers and residual writers (§D.9) is introduced post hoc to account for the observed selection and does not yield a prediction that is subsequently tested. Because §D.8 indicates that selection depends on cross-module gradient scale, the overrepresentation of V and K, whose BB matrices have far fewer rows (256 in Qwen2.5-3B), may reflect the dependence of per-entry gradient magnitude on module shape rather than a functional role. The analysis in Fig. 4 raises a similar concern. Random and informed subsets with comparable weight norm but located in different modules produce different behavioral changes, and attributing this to the "leverage of the selected parameters" restates their location rather than explaining it. In addition, dense LoRA restricted to the two top modules matches full LoRA in all six GRPO configurations (Table 35), and element-level circuits change substantially with the AA seed or the scoring objective while module and layer allocations remain stable (Table 7 at ε≥1\varepsilon\ge1, Tables 20 and 21). The control that would resolve this question is not reported, namely random entries sampled to match the circuit's allocation across modules and layers at the same kk and α\alpha. If such a baseline performs comparably to S^\hat S and F^\hat F, the finding would reduce to the statement that sparse GRPO adapters should be placed in early-layer V, K and O projections. That result would remain useful, but it would constitute a different contribution, and it would overlap with prior findings that V is the most effective projection to adapt (SMT, He et al., ICLR 2025) and that the attention output projection carries much of the reasoning behavior (Shao and Wu, 2025).

The mechanism described in the first weakness is also not connected to the scoring rule. The paper argues that circuits select coordinates whose GRPO gradients have a consistent sign, but the circuits are scored with the supervised cross-entropy loss on reference solutions (§E.5), the sign-consistency analysis in §D.9 appears to use the same loss, and the only direct comparison shows that a cross-entropy circuit shares 11 to 13% of its entries with a GRPO-scored circuit (Table 20). The statement in the conclusion that the procedure "identifies a sparse subset of B elements with consistently signed gradients" is therefore not established for the GRPO gradient, and the statement at line 258 that RL objectives select "similar circuits" overstates Table 20. The use of reference solutions for scoring also admits a simpler account of the GRPO results. Under the zero-shot prompts used in the paper, base Qwen2.5-1.5B attains 9.5% on GSM8K and 17.4% on MATH-500, considerably below published few-shot results for models of this size (the Qwen2 technical report lists 58.5% on GSM8K for Qwen2-1.5B), whereas full LoRA after GRPO reaches 63.7%. A substantial part of the improvement may therefore be attributable to output-format compliance, a form of elicitation that RLVR is known to produce on Qwen2.5 models even with weak or spurious rewards (Shao et al., 2025). A cross-entropy gradient computed on reference answers encodes the expected format directly, so informed placement may succeed in part because it incorporates supervised information about the output format that is unavailable to random placement or to full LoRA under GRPO. Finally, the comparison between base and instruction-tuned models is confounded with model family and training setup. All base models are Qwen models trained with TRL in bf16, all instruction-tuned models are Llama models trained with Unsloth in 4-bit precision with the BNPO loss (§E.3), and Qwen2.5-7B-Instruct appears in the gradient analysis (Table 17) but not in the mathematical GRPO results. The observation that random placement performs better on instruction-tuned models may therefore reflect the model family or training configuration rather than instruction tuning.

In summary, the empirical observation described in the first strength appears reliable, but the explanatory account built around it, which constitutes the majority of the claimed contribution, is not established and is in several places inconsistent with the paper's own results. Most of the experiments needed to resolve these issues are inexpensive, and they are listed in the questions below.

Questions for authors
  1. Which value of α\alpha was used for the random SFT runs in Tables 2 and 31, 32r32r as stated in Section 4 or 32 as stated in §E.4? How do the authors reconcile Table 31, in which random placement fails under SFT on GSM8K, with the claim that placement has limited effect under SFT? A matched comparison on GSM8K and MATH, using the same model, task, metric, kk and α\alpha for SFT and GRPO, would address the first weakness directly. If random placement succeeds under SFT and fails under GRPO in that setting, I would raise my score.

  2. Can the authors provide an interventional test of the noise mechanism? Specifically, does random placement under GRPO improve as the number of prompts or generations per step increases, and does random placement under SFT fail when the batch size is reduced or gradient noise is added? Please also specify what constitutes a step in Tables 1 and 17 for each regime (number of prompts, completions and tokens), the value of NN, and the fraction of GRPO groups with zero advantage at initialization for the 1.5B model. Given the comparatively aligned GRPO gradient for 7B on GSM8K in Table 17, why does random placement still fail in that setting? Do the gradient statistics remain similar when measured during training rather than only at B=0B=0?

  3. Please report random placement with entries sampled to match the circuit's allocation across modules and layers, at the same kk and α\alpha, under GRPO for at least 1.5B on GSM8K and 3B on MATH. Results for random placement restricted to V and K, random placement restricted to the early layers, and top-kk selection within each module would also be informative. A clear advantage of S^\hat S and F^\hat F over the stratified baseline would support the element-level claim in the title and would improve my assessment.

  4. The sign-consistency account and the distinction between S^\hat S and F^\hat F are not separated by the current experiments, since both scores perform similarly and select the same modules. What is the outcome of training on entries with high F^\hat F but low S^\hat S, that is, entries with large gradients of inconsistent sign? The proposed mechanism predicts that such entries should fail under GRPO.

  5. Do the GRPO-scored and Dr.GRPO-scored circuits of Table 20 perform well when trained with GRPO? In addition, what is the per-coordinate consistency of the GRPO gradient (∣mean∣/std|\text{mean}|/\text{std} across steps) on the cross-entropy circuit, compared with random entries in the same modules? This measurement would connect the scoring rule to the proposed mechanism.

  6. For the base model, random placement, the circuits and full LoRA, what fraction of GSM8K and MATH-500 outputs contain an extractable answer (#### or \boxed{})? How much of the improvement remains under few-shot prompting or lenient answer extraction? Relatedly, does scoring with the cross-entropy of the model's own correct rollouts, instead of reference solutions, still yield effective circuits? This would distinguish informed placement from the injection of supervised format information.

  7. Was the learning rate tuned for the full-LoRA baseline? Full LoRA is trained with a learning rate of 10−610^{-6} and α/r=1\alpha/r=1, whereas sparse runs use α/r=32\alpha/r=32 (Tables 26 and 27), and LoRA typically requires a larger learning rate than full fine-tuning. Since α\alpha and the learning rate are nearly interchangeable under Adam, do circuits trained with α=32\alpha=32 and a correspondingly larger learning rate reproduce the results in Table 3?

  8. Can the comparison between base and instruction-tuned models be performed within a single model family and training setup, for example Qwen2.5-7B against Qwen2.5-7B-Instruct on GSM8K and MATH using the TRL configuration of §E.2?

  9. How do LoRA-XS, VeRA, rank-1 LoRA on V, or the TinyLoRA configuration of Morris et al. (2026) perform under GRPO on these base models at 10K to 50K parameters? Please also discuss the relation to Mukherjee et al. (2025), "Reinforcement Learning Finetunes Small Subnetworks in Large Language Models", who report that RL updates a sparse subnetwork and that training only that subnetwork recovers full performance. The relation to Yin et al. (2026), which the paper cites only in support of freezing AA, also requires discussion. That study reports that extreme parameter reduction (VeRA, rank-1 adapters, IA3 and LayerNorm tuning) fails under RLVR, and concludes that RLVR exhibits a strict lower bound on expressivity. The present submission reports that 1K to 50K entries of BB match full LoRA under GRPO, a budget well below those found insufficient by Yin et al. Can the authors explain what distinguishes their setting, for example the base model family, the task, or the scaling factor α=32r\alpha=32r, and whether the vector-only methods of Yin et al. would also succeed under the authors' training configuration?

The remaining points are requests for clarification or suggestions. They do not individually determine my rating, but taken together they affect the reliability of the paper in its current form.

  1. Several design choices are justified only under SFT or are not ablated under GRPO. The comparison between training A and B jointly and training B alone (§C.1) uses Qwen3-0.6B with SFT and perplexity rather than GRPO. The runs use both r=8r=8 and r=32r=32 (Table 26), so the same kk corresponds to different fractions of BB across configurations, and no rank ablation is reported under GRPO. The mask is fixed at initialization, and periodic re-scoring during training is not compared. The synthetic experiment would be more informative if the target structure, optimizer, batch size and noise level were varied one at a time.

  2. The text-to-SQL, GPQA, AIME and AMC evaluations, Tables 11 to 13, and the gradient-structure tables appear to be based on single runs. With 30 AIME and 40 AMC problems, a single problem corresponds to 3.3 and 2.5 points respectively, so the claims in §4.4 and in the caption of Fig. 2 ("matches or exceeds Full LoRA across all settings") should be qualified. In Fig. 2(a), random placement at 50K reaches the same AIME 2024 pass@64 as F^\hat F and full LoRA, and in Fig. 2(b) it matches S^\hat S at pass@64. §B.2.2 also states that full LoRA performs best on AIME at 7B. The text-to-SQL statement that circuits remain within half a point of full LoRA holds only when the better of S^\hat S and F^\hat F is taken for each row, since F^\hat F is 2.4 points below full LoRA on Spider for the base model.

  3. On which training run are the displacement ratios in Table 16 computed? If they are computed on circuit-trained runs, the result follows trivially from the mask. If they are computed on full-LoRA runs, why do the top-scored entries move only 0.24 to 0.41 times as much as random entries under SFT, given that SFT gradients are described as directionally stable? A knockout ordered by the magnitude of the trained BB entries would also be a useful control for Fig. 3.

  4. Why does the GPQA row of Table 2 use rank 32, a setting in which full LoRA (36.6%) is below random placement (37.4%)? At rank 128 (Table 12), full LoRA reaches 50.7% and the circuits recover only 20% to 61% of its improvement. Random placement is reported as 37.4% at both 5K and 50K, which should be verified. With 198 questions in GPQA-Diamond, the binomial standard error is approximately 3.4 points, so this comparison is not conclusive in either direction.

  5. The abstract states that the selected parameters are "concentrated in the attention value projection matrices across model families and scales". In Table 4, the O projection receives 59.9% of the budget for 7B on MATH, and in Table 9 Llama-3.2-3B allocates 91.5% to Down and 6.6% to V. The V projection is overrepresented relative to its parameter share, but the budget is not concentrated there. Similarly, the statement that random placement "fails to improve over the base model" does not hold for 7B on MATH (57.4% against 53.6%, Table 3), and §B.2 states that random placement at 7B "did not learn at this scale", which is inconsistent with that entry.

  6. Several internal inconsistencies should be corrected. The caption of Fig. 3 identifies circuit knockout as red and random knockout as blue, whereas the legend and the plot show the reverse. Fig. 9 is captioned as Alpaca SFT, but §F.3 states that the 1.5B sweep uses OpenHermes. The 1.5B random curve in Fig. 9 (approximately 55% to 56% up to 50K) is consistent with neither Table 2 (61.6% at 50K) nor §F.3 (58.3% at 10K), and in the 3B panel random placement at 10K lies approximately 4 points below the base model. Line 241 refers to a 128-fold sweep of α\alpha, whereas Table 13 covers a 64-fold range. The base and full-LoRA accuracies in Table 22 differ from those in Table 3 (for example 85.2% against 87.4% for full LoRA on 7B GSM8K), and Fig. 6 implies a 3B MATH base accuracy of 49.0% against 50.6% in Table 3.

  7. Several smaller methodological points also require clarification.

    • Table 6 measures overlap with a reference circuit computed from N=100N=100 samples. If the N=50N=50 samples are a subset of the reference samples, the reported overlap of approximately 98% is inflated, and overlap between disjoint draws would be more informative.
    • The figure of 0.49% of SFT cost in Table 8 does not follow from §E.4, where 1,000 steps with 4 gradient accumulation steps imply that 50 passes correspond to approximately 1.25% of SFT. Please state the denominator used.
    • Checkpoints for MATH are selected by test accuracy (§E.2). This matters for small margins such as F^\hat F at 1K against full LoRA (62.4% against 63.5±0.563.5\pm0.5, Table 11).
    • The absolute gradient norms in Table 18 are compared across objectives with different loss scaling, so the statement that GRPO takes larger steps is not well defined, although the scale-free ratios are appropriate.
    • The reproducibility statement indicates that code will be released upon acceptance. An anonymized repository during the discussion period would help resolve several of the questions above.
8 comments

We thank the reviewer for a careful review. We ran the experiments requested.

A matched comparison on GSM8K and MATH, using the same model, task, metric, k and α for SFT and GRPO

Same circuits, k, α and evaluation as the GRPO runs in Table 3. Mean and std over 3 seeds.

PretrainedRandomŜF̂Full LoRA
1.5B GSM8K, SFT9.562.1 ± 0.361.3 ± 0.861.7 ± 0.560.5 ± 0.5
1.5B GSM8K, GRPO (Table 3)9.511.6 ± 3.165.2 ± 1.763.8 ± 2.663.7 ± 1.8
1.5B MATH-500, SFT17.443.8 ± 1.549.0 ± 0.348.3 ± 1.047.9 ± 2.4
1.5B MATH-500, GRPO (Table 3)17.419.1 ± 1.343.8 ± 0.942.5 ± 1.143.5 ± 1.7
3B GSM8K, SFT62.169.4 ± 0.371.1 ± 1.570.7 ± 0.170.3 ± 0.7
3B GSM8K, GRPO (Table 3)62.163.5 ± 1.770.8 ± 0.272.4 ± 0.871.5 ± 2.0
3B MATH-500, SFT50.657.7 ± 1.158.2 ± 1.258.1 ± 0.756.1 ± 2.0
3B MATH-500, GRPO (Table 3)50.652.2 ± 1.462.2 ± 0.863.4 ± 0.963.5 ± 0.5
7B MATH-500, SFT53.664.5 ± 2.068.2 ± 1.867.0 ± 1.366.3 ± 2.2
7B MATH-500, GRPO (Table 3)53.657.4 ± 0.974.2 ± 0.975.3 ± 0.775.1 ± 0.6

Random placement succeeds under SFT and fails under GRPO in every setting.

MATH SFT uses NuminaMath-CoT (GSM8K and MATH sources removed): SFT on the MATH train split scores below NumiaMATH for every method. We have added the table to the paper.

How do the authors reconcile Table 31, in which random placement fails under SFT on GSM8K

Table 31's SFT rows use a learning rate ten times below the default SFT learning rate of §E.4 (2 × 10⁻⁵ vs 2 × 10⁻⁴, α = 32). At that step size, random placement barely moves in 1,000 steps. With a larger effective step it succeeds: 62.1 ± 0.3 in the matched comparison above. Under GRPO, no step size helps: random placement stays at the pretrained level across a 64× range of α (Table 13).

Which value of α was used for the random SFT runs in Tables 2 and 31?

For SFT, α was chosen per model and method by a sweep. The values used:

SFT experimentsRandomŜ, F̂Full LoRA
Table 31, all rows323232
Table 2: Qwen2.5-1.5B1024102432
Table 2: Qwen2.5-3B, 7B, Llama-3.1-8B323232
Table 2: Llama-3.2-3B32102432

The α = 32r scaling in §4 applies to GRPO; we have corrected §4 and §E.4 to list these values.

Can the authors provide an interventional test of the noise mechanism? ... does random placement under SFT fail when ... gradient noise is added?

Yes. Gaussian noise on the trainable-entry gradient, 3× its norm, once per step; nothing else changed:

SFT, MATH-500Clean3× noiseChange
1.5B random43.8 ± 1.537.0 ± 1.8−6.8
1.5B Ŝ49.0 ± 0.348.8 ± 2.8−0.2
1.5B F̂48.3 ± 1.050.5 ± 0.2+2.2
3B random57.7 ± 1.155.4 ± 1.1−2.3
3B Ŝ58.2 ± 1.258.7 ± 0.8+0.5
3B F̂58.1 ± 0.758.9 ± 1.3+0.8
7B random64.5 ± 2.061.2 ± 2.1−3.3
7B Ŝ68.2 ± 1.867.1 ± 2.5−1.1
7B F̂67.0 ± 1.368.3 ± 2.2+1.3

Noise lowers random placement by 2.3 to 6.8 points; Ŝ moves by at most 1.1 and F̂ improves.

Does random placement under GRPO improve as the number of prompts or generations per step increases?

No. With 4× the completions per step (same total completions), random placement stays at the pretrained level: 49.4 at 3B MATH-500 (pretrained 50.6; 4 prompts × 8 or 1 × 32) and 54.6 at 7B (pretrained 53.6).

Please also specify what constitutes a step in Tables 1 and 17 ... and the fraction of GRPO groups with zero advantage at initialization for the 1.5B model.

One training example for SFT; one prompt with 8 completions for GRPO. Zero-advantage prompts carry no gradient and are excluded, so every GRPO step in the tables carries signal; at initialization they are 37% of 1.5B GSM8K prompts and 61% of 1.5B MATH prompts. We have added these definitions to the captions of Tables 1 and 17.

Do the gradient statistics remain similar when measured during training rather than only at initialization?

They remain near-orthogonal during training: at trained checkpoints (3B MATH, steps 400 and 3000) the consecutive-step cosine of the GRPO gradient is −0.003 to 0.031 and net/step 0.16 to 0.20, against 0.61 to 0.96 and 0.81 to 0.97 for SFT at initialization (Tables 1 and 17).

Given the comparatively aligned GRPO gradient for 7B on GSM8K in Table 17, why does random placement still fail in that setting?

7B GSM8K is the only base-model GRPO cell where the gradient is partly aligned (cosine 0.41, net/step 0.78, against net/step 0.18 to 0.34 in the other five; Tables 1 and 17), and it has the smallest gain of informed over random placement (6.0 points; Table 3). On the same model, MATH-500 has an unaligned GRPO gradient (cosine 0.00 ± 0.08, net/step 0.20 ± 0.05) and a gain of 16.8. SFT on 7B GSM8K is more aligned still (cosine 0.97 ± 0.01, net/step 0.98 ± 0.01).

Please report random placement with entries sampled to match the circuit's allocation across modules and layers ... for at least 1.5B on GSM8K and 3B on MATH. Results for random placement restricted to V and K, random placement restricted to the early layers, and top-k selection within each module

Matched: in every B tensor, as many random entries as the circuit has there. Z-scored top-k: z-score the scores within each tensor, then take the global top-k, which keeps each tensor's own ranking at roughly parameter-share allocation (at 3B MATH, V's share of the budget falls from 22.5% to 0.5%).

GRPOPretrainedRandomMatched randomZ-scored top-kŜF̂
1.5B, GSM8K9.511.6 ± 3.127.7 ± 9.410.6 ± 0.365.2 ± 1.763.8 ± 2.6
3B, MATH-50050.652.2 ± 1.454.1 ± 0.555.1 ± 0.362.2 ± 0.863.4 ± 0.9
7B, MATH-50053.657.4 ± 0.965.7 ± 1.268.7 ± 0.374.2 ± 0.975.3 ± 0.7
7B, GSM8K79.380.9 ± 1.281.4 ± 0.482.2 ± 0.486.9 ± 1.087.2 ± 1.7

Random and the circuits are Table 3. At 3B MATH-500, random placement restricted to V and K reaches 54.6 ± 0.4, and restricted to layers 0–4, 50.9 ± 0.2.

Neither the allocation nor the within-tensor ranking alone recovers the circuit. At 1.5B GSM8K the within-tensor ranking alone stays at the random level (10.6 vs 11.6); at 7B GSM8K, matching the allocation adds nothing over random (81.4 vs 80.9).

The knockout curve therefore indicates that some modules matter more than others, which the global top-k selection already encodes.

Agreed; the knockout alone cannot separate allocation from entry identity. The training controls above do. We have revised §D.8 accordingly.

What is the outcome of training on entries with high F̂ but low Ŝ

Under the cross-entropy objective such entries barely exist: among F̂'s top 10K entries the median sign consistency (Σg)²/(NΣg²) is 0.90 (random signs: 1/N = 0.02; 1.5B GSM8K), so Ŝ and F̂ share 96% of their top-10K entries and train alike (Table 3).

Do the GRPO-scored and Dr.GRPO-scored circuits of Table 20 perform well when trained with GRPO? Relatedly, does scoring with the cross-entropy of the model's own correct rollouts, instead of reference solutions, still yield effective circuits?

Yes. Every circuit of Table 20 works under GRPO (1.5B GSM8K; random placement):

Scoring objectiveŜF̂Random
Cross-entropy (Table 3)65.2 ± 1.763.8 ± 2.611.6 ± 3.1
GRPO60.162.610.6
Dr.GRPO64.661.311.1

We score with cross-entropy because it is cheaper: 50 forward-backward passes and no sampling. A circuit scored on the 3B model's own correct rollouts reaches 58.8 on MATH-500 under GRPO, vs 52.2 random and 62.2 for the reference-scored circuit (Table 3); the two share 90.6% of their top-50K entries.

what fraction of GSM8K and MATH-500 outputs contain an extractable answer? ... the Qwen2 technical report lists 58.5% on GSM8K

Published base-model numbers are few-shot; every comparison here uses the same zero-shot prompt for every experiment, including the pretrained model. The graders do not reward format: MATH-500 is read with math_verify (the pretrained 3B / 7B models already use \boxed{} in 81% / 94% of outputs), and GSM8K parses the whole output with math_verify, using #### only to truncate. Where the pretrained models already score 50.6 to 79.3 (3B, 7B; Table 3), random placement gains 1.4 to 3.8 and the circuits 7.6 to 21.7. The self-rollout circuit above uses no reference solutions.

Can the comparison between base and instruction-tuned models be performed within a single model family and training setup

Table 36 is that comparison: four Qwen2.5-7B variants, one setup, GRPO with an execution reward.

SpiderPretrainedRandomŜF̂Full LoRA
Qwen2.5-7B57.856.069.266.669.0
Qwen2.5-7B-Inst64.966.771.771.572.1
Qwen2.5-Coder-7B63.465.470.770.270.1
Qwen2.5-Coder-7B-Inst73.374.777.977.778.0

The gain of informed over random placement shrinks as the model moves toward the domain (Ŝ minus random: 13.2 on the base model, 5.0 and 5.3 on Instruct and Coder, 3.2 on Coder-Instruct), and informed placement tracks full LoRA throughout.

Since α and the learning rate are nearly interchangeable under Adam, do circuits trained with α = 32 and a correspondingly larger learning rate reproduce the results in Table 3?

As the reviewer notes, under Adam α = 32r at one learning rate and α = 32 at an r-times larger one give nearly the same update, so that run reparameterizes Table 3 rather than tests it. The question it stands in for, whether random placement fails at every step size, is answered by Table 13: random stays at the pretrained level across a 64× range of α, while the circuits improve with α.

How do LoRA-XS, VeRA, rank-1 LoRA on V, or the TinyLoRA configuration ... perform under GRPO ... the relation to Yin et al. (2026) ... Mukherjee et al. (2025)

Rank-1 LoRA on V (3B, MATH-500, 83K trainable parameters) reaches 52.0 under GRPO, no better than random placement (52.2), while the 50K-entry Ŝ circuit reaches 62.2 (Table 3). LoRA-XS, VeRA and TinyLoRA change the parameterization itself, training a few coefficients over fixed or random bases; we keep LoRA's parameterization and choose which entries of B to train, so we did not run them. Placement could be applied inside them, for example to choose which entries of the LoRA-XS core to train.

Yin et al. report that under RLVR rank-1 LoRA gains nothing over the base model, as our rank-1 V result does, and that LoRA-FA, which trains only B, matches LoRA, which supports training B alone. Mukherjee et al. find the subnetwork after RL fine-tuning, as the 5 to 30% of weights that changed; we find ours before training, from 50 forward-backward passes at initialization. We have added both to the related work.

The comparison between training A and B jointly and training B alone (§C.1) uses Qwen3-0.6B with SFT and perplexity rather than GRPO.

Training B with A frozen is the LoRA-FA configuration (Zhang et al., 2023). Under GRPO it gives up nothing in our settings: B-only circuits of 10K to 50K entries match full LoRA, which trains both A and B (Table 3). LoRA-FA has been studied in RLVR by this recent work (Yin et al. 2026) which shows that it can sometimes outperform LoRA.

no rank ablation is reported under GRPO.

At fixed rank, Tables 11 and 3 vary the budget tenfold (k = 1K, 5K and 10K, 0.005% to 0.05% of B at Qwen2.5-1.5B): random placement stays at the pretrained level at every budget while the circuits improve with k. Table 3 shows the same pattern at both ranks (r = 8 at 3B and 7B MATH and 7B GSM8K; r = 32 at 1.5B and 3B GSM8K).

The mask is fixed at initialization, and periodic re-scoring during training is not compared

The entries chosen at initialization stay useful throughout training: on the trained runs they keep moving in one direction (displacement ratio |B_T| / Σ_t |ΔB_t| of 0.82, where 1 means no back-and-forth, vs 0.59 for uniform random entries in the same tensors; 1.5B GSM8K), and the circuit matches full LoRA (Table 3). Periodic re-selection is a natural extension, and is how GaLore maintains its gradient subspace in pre-training and SFT. Under GRPO, consecutive gradients are nearly orthogonal (Table 1), so a criterion recomputed from a few steps risks switching between subsets on noise rather than tracking a real shift in importance, and it adds a recurring cost and a period hyperparameter.

Why does the GPQA row of Table 2 use rank 32, a setting in which full LoRA (36.6%) is below random placement (37.4%)? At rank 128 (Table 12), full LoRA reaches 50.7% and the circuits recover only 20% to 61% of its improvement. Random placement is reported as 37.4% at both 5K and 50K, which should be verified.

Thanks for catching this. We had a transcription issue when copying the table into the paper, and we now updated the paper text accordingly. In short, the corrected r32 results confirm your point; we now report r128 at k=5K. There, Ŝ and F̂ beat random by 12.1 and 9.9 points (paired z = 5.4 and 4.4 over the same 198 questions), while random stays at the base model. They recover 53% and 43% of full LoRA's gain (full LoRA corrected to 53.4 after the same fix), so they don't match full LoRA on this task.

Table 6 measures overlap with a reference circuit computed from N samples.

Recomputed on disjoint draws (Ŝ, GSM8K, top-10K): two disjoint 50-example draws overlap 96.5% (1.5B) and 95.0% (3B); against a disjoint 100-example reference, 96.8% and 95.6%. Table 6 now reports these.

Reviewer_Yv2oOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

In this work authors propose to use gradient statistics for LoRA B matrix parameter selection for tuning. Importantly, the authors show that previous results in this field favouring random selection hold only for an optimisation regime specific to SFT, while GRPO-like optimisation regimes benefit from gradient-based parameter masks. Synthetic examples suggest that highly correlated gradients resulting from dense supervision are not sensitive to parameter selection. While full scale experiments clearly expose the failure of random parameter selection for GRPO tuning. Additionally, the authors show that selective parameter tuning allows for better transfer learning on mathematical tasks.

Critical strengths
  1. The paper's empirical results are interesting. They determine an optimisation regime that justifies selective parameter tuning in partial LoRA, significantly improving over previously established random selection.
  2. The transfer generalisation and knockout experiments make the analysis more valuable.
Critical weaknesses
  1. The difference between fine-tuned and pretrained models is small for SFT (Table 2). There is no much headroom for full LoRA vs random placement. Showing confidence intervals or std might be important to judge the significance of the negative result.
  2. The paper lacks a theoretical argument for why random parameter subspace should lead to better results in the case of correlated gradients.
Questions for authors
  1. "Any k-element subset projects onto the dominant direction with high probability, and placement has little effect." - Could you strengthen this intuition with a mathematical argument?
  2. Could you show std for SFT (Table 2) results as well?
  3. "[your method] exceeds it on AMC, where it solves every problem at pass@64, while random placement barely improves over the base model" - Both random masks and LoRA solve AMC with 64 attempts. Only AIME 2025 exposes significant differences. Are you sure that the result description is supported by the figures?
  4. Could you show the S and F value distribution for SFT vs GRPO? Is it true that the metrics for SFT should be more uniform across parameters in comparison to GRPO?
5 comments

We thank the reviewer for their comments and suggestions.

The difference between fine-tuned and pretrained models is small for SFT (Table 2). Showing confidence intervals or std

Table 2 is a 5-seed average; per-benchmark means and standard errors are in Table 5 (§B.1), e.g. 1.5B: random 62.1 ± 0.1, Ŝ 61.9 ± 0.2, F̂ 61.6 ± 0.2, full LoRA 62.4 ± 0.2. Table 2's caption now points there. To address the small headroom, we also ran SFT where it is large (same circuits, k and α as GRPO):

Qwen2.5PretrainedRandomŜF̂
1.5B GSM8K, SFT9.562.1 ± 0.361.3 ± 0.861.7 ± 0.5
1.5B GSM8K, GRPO (Table 3)9.511.6 ± 3.165.2 ± 1.763.8 ± 2.6
1.5B MATH-500, SFT17.443.8 ± 1.549.0 ± 0.348.3 ± 1.0
1.5B MATH-500, GRPO (Table 3)17.419.1 ± 1.343.8 ± 0.942.5 ± 1.1
3B GSM8K, SFT62.169.4 ± 0.371.1 ± 1.570.7 ± 0.1
3B GSM8K, GRPO (Table 3)62.163.5 ± 1.770.8 ± 0.272.4 ± 0.8
3B MATH-500, SFT50.657.7 ± 1.158.2 ± 1.258.1 ± 0.7
3B MATH-500, GRPO (Table 3)50.652.2 ± 1.462.2 ± 0.863.4 ± 0.9
7B MATH-500, SFT53.664.5 ± 2.068.2 ± 1.867.0 ± 1.3
7B MATH-500, GRPO (Table 3)53.657.4 ± 0.974.2 ± 0.975.3 ± 0.7

Where SFT has room to improve, random placement works under SFT and fails under GRPO. (MATH SFT uses NuminaMath; SFT on the MATH train split scores below it for every method.)

Any k-element subset projects onto the dominant direction with high probability, and placement has little effect.' Could you strengthen this intuition with a mathematical argument

A sparse mask trains only the entries it selects, and under SGD, momentum or Adam an entry's net update is driven by the sum of its gradients over steps, so it makes progress only if its gradient sign is consistent across steps. If the per-step gradients share one direction u, every entry's sign is consistent and any subset S moves along u (⟨P_S u, u⟩ = ‖P_S u‖² ≥ 0); under Adam, which normalizes each entry's step, a random subset moves about as fast as an informed one. If the direction changes from step to step, most entries flip sign and their updates cancel; only a subset whose entries keep a consistent sign makes progress, and a random subset is mostly entries that cancel. Our measurements put SFT in the first case and GRPO in the second:

Gradient at initialization (Tables 1 and 17, six base-model cells)Step-to-step cosineNet / per-step norm
SFT0.61 to 0.960.81 to 0.97
GRPO−0.04 to 0.410.18 to 0.78

The trained runs show the second case directly: in the same B tensors, uniform-random entries move back and forth more than circuit entries (displacement ratio |B_T| / Σ_t |ΔB_t|: 0.59 vs 0.82; 1.5B GSM8K). Adding gradient noise to SFT moves it toward the second case and lowers random placement by 6.8 points while Ŝ holds (MATH-500: 43.8 → 37.0 and 49.0 → 48.8).

Both random masks and LoRA solve AMC with 64 attempts ... Are you sure that the result description is supported by the figures

The reviewer is right. On AMC at pass@64 (3B, trained on MATH) F̂ reaches 100.0, full LoRA 95.0, Ŝ 90.0, random 87.5, base 85.0: random does improve, and the separation at large k is small. We have rewritten the sentence to state the AMC numbers and to place the claim on AIME, where informed placement separates more clearly.

Could you show the S and F value distribution for SFT vs GRPO? Is it true that the metrics for SFT should be more uniform across parameters in comparison to GRPO

Not at initialization. We scored the same B entries with the SFT (cross-entropy) and GRPO (advantage-weighted) objectives (1.5B, GSM8K) and measured how concentrated |Ŝ| is:

Ŝ scored withTop 10K entries (0.05%)Top 1%Top 10%Normalized entropy
SFT objective2.4% of total score15.3%50.5%0.949
GRPO objective2.3%13.7%46.8%0.955

The two distributions are nearly the same; the regime difference is in how the gradient varies across steps (Table 1), not in how the score is spread across entries.

Additional experiments run during the rebuttal

  • Module × layer matched random under GRPO (3B / 7B MATH-500): 54.1 / 65.7, between random (52.2 / 57.4) and the circuits (62.2 / 74.2); both allocation and entry identity matter.
  • Gradient noise under SFT (3× its norm): random drops 43.8 → 37.0 on MATH-500, Ŝ holds 49.0 → 48.8.
  • GRPO with 4× completions per step: random stays at the pretrained level (3B 49.4, 7B 54.6).
  • Circuit scored on the model's own rollouts (no reference solutions): 58.8 under GRPO (3B MATH-500; random 52.2).

I thank the authors for their response. My main concern connected to statistical significance of the results is resolved.

The two distributions are nearly the same

Thank you for the additional analysis. I agree that similar initialization-time score distributions are not necessarily inconsistent with the proposed explanation based on temporal gradient stability.

However, I think this raises a more fundamental question about the mechanism. If GRPO gradients are substantially less aligned across optimization steps, what ensures that the parameters selected using initialization-time gradient statistics are precisely those that accumulate coherent learning signals throughout training?

Could you report (1) the top-k overlap between SFT- and GRPO-derived importance rankings and (2) the cumulative gradient coherence of the selected versus randomly sampled parameters over the GRPO training trajectory?

These analyses would help distinguish whether the effectiveness of informed placement comes from identifying temporally stable parameters or simply selecting parameters with large gradients at initialization.

We thank the reviewer for the follow-up.

(1) Overlap of SFT- and GRPO-derived rankings. Table 20 reports it: the top-10K cross-entropy circuit shares 11.4% and 12.9% of its entries with the GRPO-scored circuit in two independent runs (Qwen2.5-1.5B, GSM8K; 235× and 266× chance), and both place most of the budget on the same modules (Table 21). Both train well under GRPO: cross-entropy Ŝ reaches 65.2 ± 1.7, GRPO-scored Ŝ and F̂ reach 60.1 and 62.6 (§D.7).

(2) Cumulative coherence over the GRPO trajectory. On the trained GRPO runs (1.5B GSM8K) we measured, per trained entry, |B_T| / Σ_t |B_{t+1} − B_t| from checkpoints every 200 steps (1 = moves in one direction). It is 0.59 for uniform random entries, 0.75 ± 0.01 for random entries matched to the circuit's allocation, and 0.82 for Ŝ (§D.9). Selected entries accumulate more coherently than random ones, including random entries in the same tensors. Large versus stable gradients. Under the cross-entropy objective at initialization the two coincide: Ŝ (|Σg|, which rewards a consistent sign) and F̂ (Σg², which rewards magnitude) share 96% of their top-10K entries, because among F̂'s top entries the median sign consistency (Σg)²/(NΣg²) is 0.90 (chance 0.02). Our measurements therefore cannot separate the two readings at initialization. What they show is that the selected entries are the ones that accumulate coherently during GRPO training.

We would appreciate if the Review can adjust their scores if their concerns are addressed.

Reviewer_W4zfOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper try to answer, with A frozen and B initialized to zeros in LoRA, it matters which k entries of B are trained.

The paper conducted various experiments to compare entries are random or the top-k of a score from 50 cross-entropy forward-backward passes at initialization, S^\hat S (|mean gradient|) or F^\hat F (mean squared gradient).

The paper finds that, under SFT, random and informed placement perform comparably. Under GRPO, random placement stays near pre-trained accuracy while informed entries match full LoRA on GSM8K and MATH500.

Critical strengths
  1. This paper provides a detailed study of placement under GRPO. On Qwen base model, random placement of trainable entries in B gains only few points over the pre-trained model, while the informed (gradient-scored) placement at the same budget matches full LoRA on GSM8K and MATH500.

The contrast survives a 64× sweep of α\alpha for random placement (Table 13) and budgets down to 1K (Table 11), and the ordering repeats with a shaped execution reward on text-to-SQL for four Qwen2.5-7B variants (Table 36).

  1. The selection is cheap and simple. The selection only needs 50 cross-entropy forward-backward pass. It is stable to sampling and is a gradient mask on standard LoRA.
Critical weaknesses

The claim "which entries" is not fully delivered and we may argue "which modules/layers" is also important from author's data.

  1. From table 20-21, the cross-entropy and RL scored circuits share only 11-16% of entries, but they are agree on module.
  2. changing A moves entries, not modules/layers (§4.1).
  3. z-scoring scores within each B tensor keeps every tensor's internal ranking and changes only how many entries each tensor (one module in one layer) contributes, yet makes knockout 'indistinguishable from random at every fraction' (§D.8)
Questions for authors
  1. I suggest authors run a module×layer matched random control under GRPO in the settings of the §D.8 knockout (Qwen2.5-3B and -7B on MATH-500 at kk=50K, Table 3): sample k entries uniformly within each module and layer cell in the same counts as the S^\hat S or F^\hat F circuit, at the same α\alpha. Please show the z-scored knockout curves and how that selection splits entries across modules and layers. §D.8 reports this result only in text. Near-base accuracy would show that entry identity matters and support the title claim (score up); accuracy near the circuits would mean the result is about how the budget is allocated across modules and layers, and the framing should change accordingly.

  2. How were g1,…,gNg_1,\dots,g_N in Tables 1 obtained? independent minibatch at B=0B=0 (in which case "consecutive-step cosine" measures minibatch noise, not alignment along training), and with what per-step batch composition for SFT vs. GRPO? Could you report, for the actual sparse GRPO runs, per-coordinate sign consistency over training (∣∑tgt,i∣/∑t∣gt,i∣|\sum_t g_{t,i}|/\sum_t |g_{t,i}|) for random vs. circuit masks? Evidence that random-mask updates cancel while circuit updates do not would support the mechanism; otherwise the abstract's explanation should be withdrawn.

4 comments

We thank the reviewer for their review and suggestions.

a module×layer matched random control under GRPO in the settings of the §D.8 knockout

Matched in every module × layer cell, as many random entries as the circuit places there, at the same k and α.

GRPOPretrainedRandomMatched randomŜF̂
1.5B, GSM8K (k = 10K)9.511.6 ± 3.127.7 ± 9.465.2 ± 1.763.8 ± 2.6
3B, MATH-500 (k = 50K)50.652.2 ± 1.454.1 ± 0.562.2 ± 0.863.4 ± 0.9
7B, MATH-500 (k = 50K)53.657.4 ± 0.965.7 ± 1.274.2 ± 0.975.3 ± 0.7
7B, GSM8K (k = 50K)79.380.9 ± 1.281.4 ± 0.486.9 ± 1.087.2 ± 1.7

Random and the circuits are Table 3. At 7B GSM8K matched random stays at the random level, the outcome the reviewer associates with entry identity. In the other settings it lands between random and the circuits. In no setting does the allocation alone recover the circuit.

Please show the z-scored knockout curves and how that selection splits entries across modules and layers.

Share of the top-50K F̂ selection (%), the selection §D.8 knocks out:

QKVOGateUpDownLayers 0–4
3B, parameter share7.10.90.97.138.438.47.113.9
3B MATH, raw F̂3.22.312.813.425.024.319.018.4
3B MATH, z-scored F̂9.11.01.17.338.435.67.515.1
7B, parameter share7.21.01.07.238.138.17.217.9
7B MATH, raw F̂1.63.828.734.213.415.92.472.7
7B MATH, z-scored F̂9.11.32.515.630.524.716.49.2

Z-scoring moves the budget close to parameter share, i.e. close to uniform random's allocation, which is why at 7B its knockout tracks random. We have added this table to §D.8.

From table 20-21, the cross-entropy and RL scored circuits share only 11-16% of entries, but they agree on module ... changing A moves entries, not modules/layers

Agreed that entry identity varies across scoring objectives and A while allocation is stable. The matched control above shows allocation alone is not sufficient. We also scored a circuit on the model's own correct rollouts (no reference solutions): it shares 90.6% of the reference circuit's top-50K entries and reaches 58.8 under GRPO (3B MATH-500; random 52.2).

How were g1,..gNg_1,..g_N in Tables 1 obtained? independent minibatch, and with what per-step batch composition for SFT vs. GRPO?

Yes, at initialization on independent examples: one training example per step for SFT, one prompt with 8 completions for GRPO (zero-advantage prompts excluded). At this matched batch of one, consecutive SFT gradients align (cosine 0.61 to 0.85) and GRPO gradients do not (−0.03 to 0.04), so the difference is not the batch size. We have stated this in §3.3 and §C.

Could you report, for the actual sparse GRPO runs, per-coordinate sign consistency over training for random vs. circuit masks? Evidence that random-mask updates cancel while circuit updates do not would support the mechanism

On the trained runs we measured, per trained entry, the displacement ratio |B_T| / Σ_t |B_{t+1} − B_t| from checkpoints every 200 steps (1 = moves in one direction; lower = moves back and forth), in the B tensors both masks use (1.5B GSM8K):

Uniform randomMatched randomŜF̂
Displacement ratio0.590.75 ± 0.010.820.79

Uniform-random entries move back and forth more than circuit entries. We have added this to the appendix.

Additional experiments run during the rebuttal

  • Matched SFT vs GRPO (1.5B, same circuits, k, α): random succeeds under SFT (GSM8K 62.1, MATH-500 43.8) and fails under GRPO (11.6, 19.1).
  • Gradient noise under SFT (3× its norm): random drops 43.8 → 37.0 on MATH-500, Ŝ holds 49.0 → 48.8.
  • GRPO with 4× completions per step: random stays at the pretrained level (3B 49.4, 7B 54.6).

Dear Reviewer,

With the rebuttal window closing soon, please let us know if our rebuttal has answered your questions.

Sincerely,

The Authors.

Thanks for the reply. My main concern was whether the gain come from entries or only come from how the budget are allocated across modules/layers. The author's module x layer matched random control results addressed my concern. Therefore I raised my overall rating from 2 to 3.

We thank the Reviewer for their suggestions and for raising the score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.