- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Agent-Conditioned Routing (ACR), which assigns persistent shared/private expert-eligibility masks to different trajectories during the generation prefix of a frozen mixture-of-experts model. The goal is to increase solution-method diversity without parameter updates or assigned roles. The authors evaluate ACR on a self-constructed benchmark of 126 mathematics and physics problems using model judges, a blinded human audit, and ablation studies. The results show an increase over temperature sampling in the number of problems with at least two distinct primary methods among answer-correct trajectories, although some configurations incur a correctness loss.
Critical strengths
This work propose a simple, lightweight inference-time diversification mechanism supported by complementary evaluations. The method requires no training or additional model copies and supports parallel generation of independent trajectories. Model-based evaluation, a blinded human audit, and routing ablations jointly provide empirical support for its diversification effect.
Critical weaknesses
-
The novelty and value of the additional routing design are not sufficiently established. Routing interventions for increasing reasoning diversity have already been explored. The main additional design here is the persistent shared/private expert mask. However, the ablations show that simpler alternatives retain most of the gains, while the full construction provides relatively small additional improvements without paired uncertainty estimates. The evidence therefore does not sufficiently establish the distinctive value of this design.
-
The necessity and independent value of the self-constructed benchmark are insufficiently justified. Evaluating solution-method diversity requires derivation-level annotation, but not necessarily newly authored problems. The paper does not adequately explain why the same evaluation protocol cannot be applied to public benchmarks such as AIME or LiveCodeBench. The main results are also confined to the self-constructed, screened dataset, without systematic validation on standard benchmarks. Consequently, both the benchmark's independent contribution and the method's external validity remain insufficiently established.
Questions for authors
-
What is the contribution of the additional mask structure? In Table 7, simpler masking schemes retain most of the gains. Are the additional improvements from persistent shared/private masks stable? How should their diversity–correctness tradeoff relative to token-resampled masks be interpreted?
-
Why is a self-constructed benchmark necessary? What specific limitations of existing mathematical benchmarks prevent the same protocol from being applied to AIME or LiveCodeBench? Are there results on standard benchmarks that support the method's generalizability?
-
How do you ensure that the evaluation captures diversity of valid solution methods? Correctness is primarily determined by whether the correct answer is reached. Would the main conclusions hold when derivation validity is checked and coarser method categories are used? What is the level of agreement among the human annotators?
Thank you for requesting these controls and judging details. We completed both controls and compared GLM and Sol method counts.
W1/Q3(i): DES and activation-count control
DES varies how many experts are activated, carries that count along search paths, and uses verifier-guided selection. ACR keeps k=8 and changes which experts each trajectory can access. We will cite DES; persistence alone is not our novelty claim.
Masking can skip higher-ranked experts without changing their count: with experts ranked 1>2>3>4, ACR can select {1,3}; unrestricted top-k cannot.
Our DES-style control generates one greedy trajectory for each k∈{6,7,9,10}, with all experts eligible and each count fixed across layers and tokens. It uses the same 126 questions, prompt, repetition penalty, output cap, Sol judge, and judging prompt as ACR.
Counts are out of 126, averaged across ACR seeds. ACR yields 17 more two-method problems (60.7% more), with correctness lower by 2.11 points. This tests activation-count variation, not full DES. The control has no handoff; average k=8 does not guarantee equal compute because output lengths differ.
For arXiv-only work, ICLR guidelines say missing comparisons alone do not justify rejection. This depends on publication status; such work still informs novelty.
W1/Q3(ii): Shared/private masks and persistence
The new control independently draws 184 eligible experts per trajectory and layer at every decoding step. All four arms use top-8 routing, greedy decoding, the same prompt, 2,000-token handoff, and output cap. The new arm covers all 126 questions over three seeds with the same Sol judge and instructions.
Temperature T=0.9 gives N≥2=27.00, correctness 100.00%, and UCM@4 1.2328.
ACR yields 12.67 more two-method problems than independent resampling (39.2% more), with correctness lower by 1.78 points. This contrast represents 70.4% of ACR's 18-problem gain over temperature. Persistence helps under both constructions; shared/private masks help with or without persistence. The closest-control gap remains 2.67 problems. These comparisons are descriptive, without paired significance tests or proof that either choice is necessary.
W2/Q1: Judging and agreement
GLM and Sol ran through harnesses that spawned a subagent for each problem. Subagents read raw predictions in interactive file-reading sessions using the same instructions. Delegation alone does not establish blinding or isolation from other artifacts.
We matched available counts on identical generations: 126 questions across three seeds, or 378 paired judgments per condition.
Kappa uses linear weights. Intervals resample questions 20,000 times, keeping each question's seeds together. Lower ACR agreement shows evaluator sensitivity. These compare counts, not method identities, and do not reconstruct all submitted aggregates. One inconsistent GLM record per condition is retained; excluding it gives κ=0.387 and 0.698.
The submitted blinded 30-problem human audit provides another check with model and setting identities hidden: N≥2=12 versus 9 on Qwen and 14 versus 7 on Gemma (ACR versus temperature). We will release all available labels.
Q2: Correctness discrepancy and T=1.1 results
Expert-Sample: GLM accepts 503/504; Sol, 504/504 (99.80% agreement). They differ on the Wallis product ∏n=1∞4n2−14n2=π/2: GLM rejects the unfinished ending; Sol credits the earlier valid Stirling proof. This soft-correctness boundary also explains temperature-arm gaps.
For T=1.1, Sol's three-seed means are N≥2=29.00, N≥3=2.33 (out of 126), Correctness =99.60% and UCM@4 =1.253.