- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies the allocation of work budget in random-feature PDE solvers. The two strategies HybridWide and Gate-RFM+ (one focuses on a wider trial space and the other focuses on several smaller candidates with physics-based selection among them) are compared under the same feature distribution and fitting process. The main experiments on 60 linear advection-diffusion-reaction problems show that the search policy is worse than the policy with a wider model. An oracle analysis also indicates that most of the gap is present in the generated candidate sets.
Critical strengths
-
The paper studies an interesting and practically relevant problem: how to allocate limited computation in random-feature PDE solvers. The results offer insights into the trade-off between increasing model capacity and searching over more candidates.
-
The experiments are extensive, comparing several allocation strategies under different budgets and including detailed ablation studies.
Critical weaknesses
My main concerns are the following two points:
-
The significance of the contribution is not very convincing to me. The question itself is interesting and meaningful, and the paper does provide a detailed analysis of why the search policy performs worse than a wider model. However, even with the additional evaluations, it remains unclear how far these findings extend beyond specific solver designs and allocation rules studied in the paper. As a result, while the negative result is useful, I'm not sure that it provides substantial new understanding of compute allocation.
-
The experimental design feels somewhat ad-hoc, particularly how the work budget is defined and allocated between fitting and verification. It's not very clear to me how robust the conclusions are to different cost assumptions or allocation rules.
Also, the presentation introduces many implementation details without making their relevance to the central conclusion clear, which is not very friendly to first-time readers.
Questions for authors
-
Could the authors clarify what broader insights about compute allocation can be established from the results? And which of them depend on the specific experimental settings?
-
How were the costs of fitting and verification determined?
2 comments
Thank you for the detailed response and additional experiments. The cost-sensitivity analyses and clarifications address much of my concern about the experimental setup. However, I still find the broader significance unclear: the findings remain closely tied to the particular solver designs and allocation rules studied, and I am not convinced that they offer enough new understanding beyond these settings. I appreciate the authors’ efforts, but I will maintain my original rating of weak rejection.
Official Comment by Authors
Thank you for your careful review. Your concerns about significance, cost assumptions and presentation led us to add direct comparisons and clarify the contribution. All locations refer to the revised manuscript.
W1 & Q1: Broader insights and dependence on the setting. We agree that the experiments do not establish a universally optimal allocation. We now emphasize three conditional lessons (Sec. 6, p. 10, Appendix H.4, p. 41).
First, candidate quality and final selection require separate evaluation. For a completed population with positive errors, selected/wide equals (oracle/wide) × (selected/oracle). The primary geometric ratios are 9.318, 9.255 and 1.0068. Improving only final selection cannot repair those populations. Intermediate promotion, fitting or candidate construction can still change them. This is an algebraic diagnostic requiring reference-error auditing, not a new approximation theorem or an online controller for unknown solutions (Sec. 3.3, p. 4).
Second, useful scoring need not justify the allocation producing the candidates. Random final choice is 255-fold worse within the same completed populations despite their small remaining oracle gap. This separates scoring value from population-generation value (Sec. 5.3, p. 8).
Third, width/diversity competition depends on construction and costs. With N_f=8M and no held-out evaluation, an unrounded fit costs 24M^2+(40/d)M^3. Ideal equal one-off splits reduce width as K^(−1/2) or K^(−1/3) in the respective dominant-term regimes, with crossover near M=3d/5. This is conditional cost algebra, not a hardware or accuracy law (Appendix H.2, p. 36). Periodic reversals and enrichment results demonstrate why the numerical ordering cannot extend to all solvers or PDEs. The finite-set ranking and restricted elliptic bounds do not ensure competitive candidates exist.
W2 & Q2: Cost origin and allocation robustness. We appreciate that the original convention needed explanation. The inherited 1/64 scale kept operator and linear-algebra terms visible at pilot widths. That pilot used a different leading-QR expression and does not derive or calibrate the later proxy. We now distinguish these conventions and measured online time (Appendix H.1, p. 35).
We reran the action cost as floor(3N_fM + (4N_fM^2+8M^3)/d + v(N_s+N_a)(3M+K_v)), with arithmetic denominator d in {16,64,256} and held-out price v in {0.25,1,4}. Search uses N_f=max(256,8M), N_s=256, N_a=512 and K_v=32. Wide has N_s=N_a=0. Default prices are d=64 and v=1. Smaller d makes arithmetic more expensive, including enrichment screening. Scores and evaluation sizes stay fixed. Every promotion pays a full refit, acceptance work is reserved, and fresh acceptance points follow population completion. Reference errors never select an online action.
The five policies are original ladder-wide/search, integer-width filled-wide, filled-search and frozen E64. Filled-wide takes the largest affordable integer width. Filled-search preserves the original search prefix and uses remaining work for wider prefixes or new banks chosen by search scores. All actions are charged. Width caps and action sizes can still leave unequal unused work, so this is a common-ceiling comparison, not equal wall time.
All 38,880 distinct runs completed on the same 60 PDEs, six budgets and four seeds without failures or overspending. We reran affordability decisions rather than relabeling errors. Default prices reproduce 4,320 earlier wide/search/E64 paths, and every filled-search prefix matches. Inference averages paired logs within PDE. Marginal 95% intervals resample the 60 PDEs, with a separately declared 441-contrast global Holm correction (Appendix H.2, p. 36).
Across nine pooled price cells, selected/wide is 2.92–140, oracle/wide 2.88–139, and filled-search/filled-wide 4.39–291. The selected and oracle intervals all exceed one, with Holm-adjusted p=0.00882. Individual tasks and budgets can differ. E64/filled-wide is 0.344 (0.245–0.486), 0.929 (0.647–1.315) and 4.16 (2.99–5.76) at d=16,64,256. Its default advantage is inconclusive, while the opposite effects at 16 and 256 both have adjusted p=0.00882 (Table 5, p. 9, Table 37, p. 38). All budgets and adverse outcomes remain in Appendix H.3 (p. 37). These finite checks reuse tasks and do not calibrate arbitrary hardware costs. Earlier verification-quota and deadline studies remain separate (Appendix F.3, p. 28).
Presentation. Thank you for this suggestion. The Introduction now foregrounds allocation diagnosis. Sec. 5.3 (p. 8) links ablations to representation, generation or selection. Table 5 (p. 9) summarizes costs and allocation, while Appendix H.5 (p. 41) retains every component outcome, including null and adverse results. The revised main text remains 10 pages.
Thank you again for helping us clarify the broader insights and their limits.