- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper treats parent selection in LLM-guided test-time discovery (TTT-Discover-style search) as a batch allocation problem. DFT scores a parent by how its past best-of-n bundle compares with the best of n historical offspring. LC adds a submodular ancestry-coverage term that discounts parents from lineages already represented in the batch, with a weight annealed from 1 to 0. With Qwen3-8B, LC-DFT improves mean scores on the math construction tasks, AHC058 and denoising, while DFT alone is best on GPU kernels. With Qwen3.8-27B, the paper reports a new Circle-32 record and fast kernels.
Critical strengths
- The batch view of parent selection is natural, and the tail-integral formulation of expected discovery gain is clean and correct.
- The method is cheap and easy to drop into existing pipelines.
- The paper is unusually transparent: it reports that LC hurts on kernels, and documents repairs, timeouts and protocol differences in the appendix.
- The search-tree analysis (Fig. 2, Table 10) shows what the selector actually does.
Critical weaknesses
- Relation to the baseline. TTT-Discover's PUCT already blocks a selected parent's ancestors and descendants from the same batch, and backpropagates visit counts to all ancestors. LC reads more as a soft version of this existing lineage mechanism than as a new idea, and the paper doesn't discuss it. The DFT objective is also closely related to max-k-armed / extreme bandits (Cicirello & Smith 2005; Carpentier & Valko 2014), which aren't cited.
- Theory vs. algorithm. The derivation motivates a submodular set objective, but the method uses a sum of singleton scores plus a hand-designed ancestry proxy. Also, the surrogate (Eq. 7) weights thresholds below the current frontier, so it measures "better than a typical bundle" rather than "likely to advance the frontier". The key assumption, that shared ancestry predicts correlated offspring outcomes, is never tested, although the logged bundles would allow it.
- Statistical support. The 8B gains are small (e.g., about 2e-5 on Erdős, about 1% on AHC058), with 3 runs and no error bars. Circle packing uses single runs, with gaps at the 1e-12 level, the same as the verifier tolerance. AC1/AC2 compare in-house means against single published PUCT numbers. The Tabula gap comes from one PUCT timeout scored as zero.
- Overclaiming. The Circle-32 "new state of the art" (2.939572771206 vs. 2.939572771205) differs in the last displayed digit, after a radius repair step, so I would call it a tie. The 27B results have no 27B baselines, so they don't isolate the method's contribution. The MLA comparison re-times TTT-Discover kernels on H200, although they were selected on MI300X. The abstract says the method works with a fixed model, but all runs use test-time LoRA updates.
Questions for authors
- Does your PUCT reproduction include TTT-Discover's lineage blocking? How does LC differ in effect?
- Please add std or per-run values, run LC on top of PUCT, add a "same λ schedule, simpler diversity rule" baseline, and test whether shared ancestry predicts correlated bundle outcomes.
- Why is AHC039 missing from the 8B comparison, and why does LC hurt on kernels?
5 comments
[Authors Response (2/3)]
4. Repeated 8B comparisons and updated denoising [W3, Q2]
Both Circle problems now have three runs per method. Below are three-run means and sample standard deviations at 50 steps; per-run results are in the revised appendix.
\text{Problem} & \text{PUCT} & \text{DFT} & \text{LC-DFT}\\\\ \hline \text{Erdős}\downarrow & 0.38096841\pm0.00005043 & 0.38094456\pm0.00001301 & 0.38092384\pm0.00000725\\\\ \text{AHC058}\uparrow & 761{,}532{,}973\pm7{,}777{,}118 & 758{,}789{,}165\pm4{,}968{,}803 & 766{,}321{,}006\pm6{,}905{,}888\\\\ \text{Circle 26}\uparrow & \text{--} & 2.635417627248\pm9.74\times10^{-4} & 2.635983084918\pm8.44\times10^{-15}\\\\ \text{Circle 32}\uparrow & \text{--} & 2.939572771206\pm1.26\times10^{-14} & 2.939572771206\pm7.22\times10^{-15}\\\\ \hline \end{array}$$ LC-DFT leads on Erdős, AHC058, and Circle-26 means; Circle-32 ties. Erdős variability is also lower. For AC1 and AC2, we retain the bounds TTT-Discover reports from one run rather than replace them with a potentially weaker reproduction. We set a stricter bar for LC-DFT than a best-run comparison: its mean over three runs must beat the published bound. It does so on both tasks. Denoising was rerun three times per method to match the generation budgets of TTT-Discover and our other problems; the appendix budget table is updated: $$\begin{array}{lrr} \text{Method} & \text{PBMC}\uparrow & \text{Tabula}\uparrow\\\\ \hline \text{PUCT} & 0.69 & 0.72\\;(2/3)\\\\ \text{DFT} & 0.70 & 0.73\\;(2/3)\\\\ \text{LC-DFT} & 0.70 & 0.73\\;(3/3)\\\\ \hline \end{array}$$ Tabula averages completed evaluations (counts in parentheses), no longer assigning zero to timeouts. LC-DFT matches DFT and completes all three. **5. Mathematical bounds and the Circle-32 record [W3, W4]** Erdős minimum overlap concerns a bound on a mathematical constant, not average benchmark accuracy. Published advances establish the scale of progress: $$\begin{array}{lrrr} \text{Study} & \text{Previous bound} & \text{New bound} & \text{Reduction}\\\\ \hline \text{AlphaEvolve [1]} & 0.380927 & 0.380924 & 3.0\times10^{-6}\\\\ \text{TTT-Discover [2]} & 0.380924 & 0.380876 & 4.8\times10^{-5}\\\\ \text{SimpleTES [3]} & 0.380871 & 0.380868 & 3.0\times10^{-6}\\\\ \hline \end{array}$$ On Erdős, 8B DFT improves the mean bound over reproduced PUCT by $2.385\times10^{-5}$; LC adds $2.073\times10^{-5}$, for a total $4.457\times10^{-5}$. Both contributions are on the scale of the published advances above. Circle-32 search uses finite-precision arithmetic. After search, we keep centers fixed, reduce radii to eliminate residual constraint violations, and verify containment and non-overlap using exact rational arithmetic with **zero tolerance**. Radius repair only decreases the objective, and the final value is rounded downward. **The resulting bound, 2.939572771206, strictly exceeds Packomania's 2.939572771205: it is a new state of the art, not a tie.** Even a last-digit improvement tightens the known constructive bound.[Authors Response (3/3)]
6. Completed 27B results and GPU evaluation [W4]
8B repetitions compare methods; 27B uses one run per problem, costing eight H200 GPUs for 3-6 days.
\text{Problem} & \text{Best literature (120B)} & \text{Ours (27B)} & \text{vs. literature} & \text{Budget ratio}\\\\ \hline \text{Erdős}\downarrow & 0.380868^{\mathrm{S}} & 0.38086449 & \checkmark & 0.5\times\\\\ \text{AC1}\downarrow & 1.50287^{\mathrm{T}} & 1.50291954 & & 1\times\\\\ \text{AC2}\uparrow & 0.962694^{\mathrm{S}} & 0.95854 & & 0.5\times\\\\ \text{Circle 26}\uparrow & 2.635983^{\mathrm{S}} & 2.635983084917 & = & 0.5\times\\\\ \text{Circle 32}\uparrow & 2.939572^{\mathrm{S}} & 2.939572771206 & \checkmark & 0.5\times\\\\ \text{TriMul}\downarrow\\;(\mathrm{H200},\\,\mu\mathrm{s}) & 1020.0^{\mathrm{S}} & 881.7 & \checkmark & 0.5\times\\\\ \text{MLA}\downarrow\\;(\mathrm{H200},\\,\mu\mathrm{s}) & 1876.9^{\mathrm{T}} & 1188.8 & \checkmark & 1\times\\\\ \text{AHC039}\uparrow & 567{,}062^{\mathrm{T}} & 575{,}374 & \checkmark & 1\times\\\\ \text{AHC058}\uparrow & 848{,}414{,}228^{\mathrm{T}} & 848{,}449{,}469 & \checkmark & 1\times\\\\ \text{PBMC}\uparrow & 0.71^{\mathrm{T,S}} & 0.71 & = & 1\times\\,/\\,0.5\times\\\\ \text{Tabula}\uparrow & 0.74^{\mathrm{S}} & 0.74 & = & 0.5\times\\\\ \hline \end{array}$$ $\checkmark$: improvement; $=$: tie at reported precision. T: TTT-Discover; S: SimpleTES. Budget ratio: ours/reference. MLA uses H200 re-evaluation; AHC results are verified by AtCoder. LC-DFT matches or surpasses the best literature results on eight of ten problems, using the same per-run rollout budget as TTT-Discover and half SimpleTES's default budget. TTT-Discover searches MLA on H200 but selects final programs on MI300X. We tested all three released programs unchanged with our H200 evaluator: $$\begin{array}{lrc} \text{TTT-Discover program} & \text{H200 runtime }(\mu\mathrm{s})\downarrow & \text{Correctness}\\\\ \hline \text{Program 1} & 2976.8 & \text{Pass}\\\\ \text{Program 2} & \mathbf{1876.9} & \text{Pass}\\\\ \text{Program 3} & 1901.6 & \text{Pass}\\\\ \hline \end{array}$$ **We use the fastest released program as the baseline.** We lack MI300X and H100 access and will consider renting them for further tests. On H200, SimpleTES [3, Supplementary Table 13] reports TTT-Discover at $1064.0\\,\mu\mathrm{s}$ and SimpleTES at $1020.0\\,\mu\mathrm{s}$, a $44.0\\,\mu\mathrm{s}$ gain. LC-DFT reaches $881.7\\,\mu\mathrm{s}$: a further $138.3\\,\mu\mathrm{s}$, over three times that gain, with half the default SimpleTES rollout budget. **7. 8B GPU behavior and missing AHC039 results [Q3]** From revised Table 10: LC-DFT minus DFT, three-run means in percentage points. The threshold is the pooled median bundle-best score; the last column compares mean scores. $$\begin{array}{lrrc} \text{Problem} & \text{Valid rate} & \text{Median-threshold} & \text{Score}\\\\ & \text{(pp)} & \text{bundle rate (pp)} & \text{vs. DFT}\\\\ \hline \text{TriMul} & -5.14 & -22.17 & \times\\\\ \text{MLA} & -8.47 & -26.67 & \times\\\\ \hline \end{array}$$ LC yields fewer valid programs and strong bundles. Our interpretation is that these problems strain the 8B model: broader exploration needs more successful refinements than the current budget supplies. More sampling could realize delayed gains, as illustrated in Figure 3; the 27B GPU results support the full method's discovery capability with a stronger generator. AHC039 did not improve over its starting solution with the 8B budget; the 27B result exceeds the listed literature reference. **8. Applicability with a fixed model [W4]** Qwen3-8B, three-run means at 50 steps; only model updates differ between the two LC-DFT rows. $$\begin{array}{llrr} \text{Method} & \text{TTT} & \text{Erdős}\downarrow & \text{AHC058}\uparrow\\\\ \hline \text{PUCT} & \text{Yes} & 0.38096841 & 761{,}532{,}973\\\\ \text{LC-DFT} & \text{No} & 0.38093988 & 761{,}634{,}453\\\\ \text{LC-DFT} & \text{Yes} & \mathbf{0.38092384} & \mathbf{766{,}321{,}006}\\\\ \hline \end{array}$$ Fixed-model LC-DFT has a better mean than trained PUCT on both tasks; training further improves LC-DFT on both. **References** [1] Novikov et al. (2025). [AlphaEvolve](https://arxiv.org/abs/2506.13131), Appendix B.5. [2] Yuksekgonul et al. (2026). [Learning to Discover at Test Time](https://arxiv.org/abs/2601.16175). [3] Ye et al. (2026). [SimpleTES](https://arxiv.org/abs/2604.19341). Inman and Bradley (1989). [Overlapping coefficient](https://doi.org/10.1080/03610928908830127). Cicirello and Smith (2005). [The Max K-Armed Bandit](https://www.cicirello.org/publications/cicirello2005aaai.html). Carpentier and Valko (2014). [Extreme Bandits](https://misovalko.github.io/papers/carpentier2014extreme.html).
[Authors Response (1/3)]
Thank you for the detailed comments.
1. PUCT reproduction and LC-PUCT [W1, Q1, Q2]
Our PUCT reproduction retains every component of the released selection procedure: descending PUCT ordering, ancestor/descendant exclusion, and propagation of visit counts to ancestors. LC instead optimizes quality-weighted batch marginals, accounting for overlap even between parents that are neither ancestors nor descendants of one another.
LC-PUCT uses the same allocator and sigmoid schedule with PUCT scores normalized by rank or min-max. Entries are three-run Qwen3-8B means after 50 steps.
\text{Method} & \text{Erdős}\downarrow\\\\ \hline \text{PUCT} & 0.38096841\\\\ \text{LC-PUCT (rank)} & 0.38094334\\\\ \text{LC-PUCT (min-max)} & 0.38093868\\\\ \text{DFT} & 0.38094456\\\\ \text{LC-DFT} & \mathbf{0.38092384}\\\\ \hline \end{array}$$ LC improves both PUCT and DFT, with LC-DFT achieving the best mean bound. The requested same-schedule comparison with a simpler diversity rule remains outstanding. **2. Quality-weighted allocation and the discovery objective [W1, W2]** The coverage term itself depends on quality. We show that expected discovery gain is monotone submodular; LC preserves this structure with quality-weighted coverage. Eqs. (11) and (17) compare the historical-threshold marginal and LC: $$\text{Eq. (11):}\quad \overline{U}_t(S\cup\\{s\\})-\overline{U}_t(S)=q_t(s)[1-o_t(s,S)].$$ $$\text{Eq. (17):}\quad \Delta_t(s\mid S)=q_t^{\mathrm{DFT}}(s)[1-\lambda_t\eta_t(s,S)].$$ Here $o_t$ is outcome overlap and $\eta_t$ is structural saturation. Quality enters both terms: $$J_t(S)=(1-\lambda_t)\sum_{s\in S}q_t^{\mathrm{DFT}}(s)+\lambda_t\sum_v\max_{s\in S}\bigl[q_t^{\mathrm{DFT}}(s)\phi_s(v)\bigr].$$ **The coverage term therefore rewards additional, high-quality opportunities rather than ancestry diversity alone.** Greedy has the standard $1-1/e$ guarantee for $J_t$; the semantic analysis below supports the structural proxy. Direct frontier-crossing scores were too sparse during development. Historical best-of-n thresholds, including those below the frontier, give DFT a dense signal distinguishing strong bundles before another record is found. **We added both suggested bandit references to Related Work and its extended appendix,** relating their best-observed-reward objective to ours and distinguishing their extreme-value models from DFT scoring and LC allocation. **3. Lineage and semantic redundancy [W2, Q2]** Figure 3 in the revised manuscript connects Figure 2's search trajectory to program semantics. We analyzed all 473 valid offspring from the DFT and LC-DFT batches at step 24 of Erdős repB, with GPT-6-Astra assistance, grouping programs by their starting construction and search strategy. DFT's inherited constructions formed one group, versus seven for LC-DFT. Across all parent pairs, overlap between offspring semantic-group distributions fell from 0.301 to 0.158, measured by the overlapping coefficient $\sum_g\min(p_g,p'_g)$ (Inman and Bradley, 1989). Distinct LC construction directions also contributed to distinct reward peaks (Figure 3(c)). Thus the lineage difference corresponds to semantic diversity in both selected parents and generated offspring. The retained directions also contributed to later discoveries. LC kept exploring a 100-height construction outside the top eight DFT ranks; at step 34, a rank-18 parent produced all three frontier-improving offspring. Unlike the leading mirror-symmetric branch, it allowed heights to vary independently. Descendants retained this construction until splitting its segments into 200 independently adjustable heights. Reusing the resulting parent at steps 42 and 43 first improved the parent, then lowered the frontier by $1.48\times10^{-5}$. This direction ultimately produced the run's best construction. Figure 3(d) and Appendix C.1 document the classifications and search path.