- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies whether additive models capture interactions among sparse routing choices. Across several settings, pairwise models improve hindsight route selection over unary models, though the results do not establish better deployment efficiency or generation quality.
Critical strengths
● The paper addresses a relevant question for sparse attention and separates temporal aggregation, interaction structure, choice regret, and held-configuration prediction rather than relying on a single fit metric. ● The estimands are clearly defined. The paper distinguishes functional ANOVA on the full Cartesian space from fitting on a constrained feasible set, and explains the scope of its uniform measure and hindsight regret. ● The experiments cover multiple models, datasets, support constructions, and target horizons. Some studies use separate cohorts, and the paper reports both errors corrected and errors introduced by pairwise terms.
Critical weaknesses
● Functional ANOVA is an established tool, and the main contribution appears to be its application to specific routing surfaces and the empirical value of pairwise terms. The paper should clarify what new insight it adds beyond prior interaction analyses and routing interventions, and which findings may generalize beyond the tested settings. ● The main hindsight results fit and select routes after observing a complete or large candidate set, so they do not show how to find better routes with less computation. The prospective calibration study improves KL fidelity relative to an anchor, but does not establish superiority over direct search or confirm gains in NLL or generation quality.
Questions for authors
● When the full routing surface is unavailable at deployment time, which results in this paper can concretely guide candidate-route acquisition or screening? Does the prospective study support a specific candidate-sampling strategy? ● Could the main text include a summary table listing each confirmatory claim, its estimand, independent statistical unit, cohort size, absolute effect, and confidence interval?
Response to Reviewer RNxK
Contribution and generalization
We use established fANOVA as a measurement tool. The new 128-document held-choice test shows that removing a predicted interaction raises regret while preserving the local mean and average main effects. The temporal/support studies measure order-dependent retention and support-recipe effects; Appendix V separates retention, Unary-relative gain and absolute selected loss.
Domain/checkpoint tests cover NLP-paper and Python-repository text and Qwen3-32B (Section 4.6, Table 6), alongside the native Llama/Qwen panels (Section 4.2). These extensions concern continuation losses and route-choice regret in the tested settings.
Candidate acquisition or screening when the full surface is unavailable
Your question calls for guidance that specifies what information a selector actually has. The concrete guidance is to fit on paid observations, validate the actual recommendations on withheld candidates, compare strong Direct and Static controls, and count calibration and controller costs. A policy should retain its best measured candidate under the same objective.
The held-choice result uses 36 fitting labels. The separate 1,280-document study tests a fixed 8+5 acquisition rule; its gain intervals against Direct and matched batch Unary include zero. Appendix Q shows an anchor-relative KL benefit for its fixed one-guide-token acquisition and four-token selection policy; it fits no P2 model and leaves the Direct comparison unresolved. These studies provide evidence for evaluating screening decisions; a generally superior, inexpensive candidate-sampling rule remains open.
Confirmatory claim summary
Table 1 maps the principal claims to their estimands, independent units, cohort sizes, effects and individual 95% intervals. We have now added the three absolute NLL RMSE reductions below to its C1/C2 rows; the original relative intervals and confirmation decisions remain unchanged.
These are added post hoc descriptive intervals from the original 20,000 paired, corpus-stratified crossed document/action bootstrap draws, with frozen fits and prior. C1 and C2 use separate cohorts (24 documents per corpus) in the prespecified Llama-interleaved NLL condition. C1 averages document residuals before RMSE; C2 calculates RMSE within each document before averaging. We reproduced the original relative intervals and independently checked the absolute intervals by direct index resampling (Apps. K/L).
Table 1 remains a principal-claim map. The study appendices retain the remaining prespecified comparisons and their original decisions, including unmet criteria. Appendix Y, Tables 74-76, gives all five held-quartet contrasts and document-paired inference; Table 6 gives domain/checkpoint scope. Appendix B records the study populations. Shared documents, repeated checkpoints, quartets and controls are not counted as additional independent observations; opened readbacks and secondary analyses retain their stated status.