acceptodds
Under review as a conference paper at ICLR 2027

FEED: Feedback-Guided Self-Training and Preference Optimization for Efficient Test-Time Discovery

Abstract

Large language models (LLMs) increasingly support scientific discovery through test-time learning, but generating candidate solutions is costly, and high-quality examples remain scarce. To improve learning under limited computational budgets, we introduce FEED (Feedback-Enhanced Efficient Discovery), which turns evaluation feedback into a unified objective for quality-weighted self-training and prompt-matched success–failure preference optimization. FEED draws valid programs from current and historical evaluations, assigning each a uniform base weight and additional emphasis according to solution quality. The same weights scale preference supervision against failures generated under the same original prompt, while every selected valid program retains direct supervision, including those without matched failures. Evaluated with Qwen3-8B on six tasks spanning mathematical construction, heuristic design, and function optimization, FEED achieves earlier discovery of high-quality solutions on multiple tasks than strong baselines, including TTT-Discover, under matched one-hour budgets covering generation, evaluation, and learning. For example, on CVRP-ACO, FEED reaches the 10.2 route-cost threshold in all five runs versus none for TTT-Discover, reducing the one-hour restricted mean time to threshold from 60.00 to 17.62 minutes. Further circle-packing evaluations show efficiency gains across Qwen3 model sizes and solution-quality improvements in two other model families. Together, these results demonstrate that FEED's feedback-guided joint optimization of self-training and success–failure preferences enables earlier discovery of high-quality solutions under fixed test-time budgets.

Primary area:
foundation or frontier models, including LLMs
Keywords:
program discovery, test-time learning, large language models, evaluation feedback, preference learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_J21HOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces FEED, a test-time training method for scientific discovery. Built upon TTT-Discover's PUCT search, FEED replaces its entropic objective with quality-weighted self-training on valid programs plus a DPO-style loss that prefers valid over invalid programs from the same prompt. Using Qwen3-8B and a one-hour budget on six tasks, FEED reaches target scores faster than TTT-Discover and OpenEvolve. Ablations show that the joint objective outperforms its single-objective variants mainly on CVRP-ACO and AC2, and that parameter updates speed up discovery over a fixed-model PUCT baseline on most but not all tasks.

Critical strengths
  • The evaluation uses a wall-clock budget that includes generation, evaluation, and training time, with a detailed time breakdown, which makes the comparison between learning-based and search-only methods fair.
Critical weaknesses
  • FEED keeps TTT-Discover's PUCT search unchanged and only replaces its learning objective. Each component of the new objective has been explored in prior work, including reward-weighted self-training (e.g., AWR [1], ReST [2]), same-prompt valid-invalid preference learning (e.g., EvoTune [3]), and joint SFT + DPO learning (e.g., Iterative RPO [4]). The paper itself notes that EvoTune suggested combining self-training and preference learning as future work. The contribution is therefore mainly a specific combination of existing objectives.
  • FEED shows a clear advantage on only two of six tasks (CVRP-ACO and AC2) compared against TTT-Discover, and the differences on the other tasks are small. On the remaining tasks, the evidence is much weaker. On AC1, the two methods are practically identical in both hits and RMST. On 32-circle, FEED reaches the threshold in one more run, but the paper notes that the mean trajectories end at similar scores. On MaxDet, the difference comes down to a single run of FEED reaching the threshold. On OBP, FEED's final mean score is lower, yet both methods reach the threshold in only one of five runs, and Table 11 and Figure 6 suggest that the gap in the mean is driven largely by one seed. Moreover, the hit and RMST results in Table 1 depend on a single threshold per task, and the paper does not explain how these thresholds were chosen.
  • TTT-Discover is not evaluated under its original setting (512 rollouts per step for 50 steps). Instead, it is run with 32 rollouts per step under a one-hour budget, while keeping hyperparameters such as the learning rate unchanged. As a result, FEED's advantage is shown only in this low-budget regime, and even within this regime, TTT-Discover may underperform because its configuration was not adapted.

[1] Peng et al., Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning, arXiv 2019
[2] Gulcehre et al., Reinforced Self-Training (ReST) for Language Modeling, arXiv 2023
[3] Surina et al., Algorithm Discovery With LLMs: Evolutionary Search Meets Reinforcement Learning, CoLM 2025
[4] Pang et al., Iterative Reasoning Preference Optimization, NeurIPS 2024 [4]

Questions for authors
  • All methods here run generation and evaluation in sequence, but search-only methods like OpenEvolve can run them in parallel, since generation uses the GPU and evaluation uses the CPU. On 32-circle, for example, PUCT spends about 39 minutes generating and 21 minutes evaluating (Table 10), so running them in parallel could let it evaluate many more candidates in an hour. Did you try this? I wonder whether FEED would still be faster in that case.
  • The preference loss only uses failed programs as negatives, and low-scoring valid programs are not treated as failures. I understand that quality differences among valid programs are partly handled by the SFT weights, but those weights are all positive, so weaker valid programs are never pushed down. Did you try using low-scoring valid programs as negatives?
  • FEED changes several things relative to TTT-Discover, not only the loss. It also reuses past programs for training and filters updates with a KL check. The ablations only test the loss terms, so I could not tell which change actually leads to the gains. Have you tried adding the historical pool or the KL-based update control to TTT-Discover?
5 comments

Thank you for the detailed explanation and additional experiments. I still have some remaining concerns as follows:

  • Novelty: A good paper does not need to contain a radical algorithmic contribution. A paper that substantially improves performance by combining many small improvements can also be a good paper. However, I don’t think your paper falls into the former category. If so, I believe your paper should demonstrate a clear practical contribution. However, the paper show performance gains only under a limited budget, and there is no justification for how this budget was chosen. Also, there is no evidence that this gain maintains with larger budgets.
  • Evaluation: In Table 11, the gap between FEED and TTT-Discover looks statistically insignificant. Also, TTT-Discover performs best at the smallest learning rate on 32-circle and CVRP-ACO, so a lower rate might help further.
AuthorsOpenReview ↗

Q1. Practical contribution and evaluation budgets

Thank you for emphasizing the importance of practical contribution. We chose a common one-hour horizon to assess discovery efficiency under limited resources and make repeated comparisons across six tasks feasible within our available compute.

Our evaluation also includes 180-minute experiments on 32-circle across multiple model sizes and families (Section 3.3.2). Table 4 reports RMST reductions relative to TTT-Discover of 34.7%, 9.8%, and 17.6% with Qwen3-8B, Qwen3-14B, and Qwen3-32B, respectively. Figure 5 additionally shows higher mean best-so-far scores than TTT-Discover at the three-hour endpoint with both Ministral and OLMo. These results provide evidence of discovery-efficiency and solution-quality benefits beyond the one-hour main setting.

FEED’s methodological contribution changes how evaluated programs enter the learning objective. TTT-Discover derives updates from entropic reward advantages; even a valid program can receive a negative reward-derived advantage when its reward is low relative to other candidates. FEED gives every selected valid program a positive self-training term and adds explicit preference comparisons against failures from the same prompt. Shared quality weights control both terms, while historical and unpaired valid programs remain reusable positive examples. The component ablations and augmented TTT-Discover comparisons in Table 12 of our previous response provide empirical support for this learning design.These experiments support FEED’s methodological contribution and practical value across the evaluated settings.

AuthorsOpenReview ↗

Q2. Statistical evidence and learning-rate coverage

Thank you for following up. The original CVRP-ACO comparison provides statistical support for FEED’s improvement: the mean one-hour cost difference is −0.0881 (FEED minus TTT-Discover), with a 95% pointwise Welch confidence interval of [−0.1464, −0.0298]. FEED also achieves 5/5 hits versus 0/5 for TTT-Discover. Broader practical benefits appear on 32-circle and AC1, where FEED achieves 5/5 hits versus 0/5 for OpenEvolve. In Table 11, FEED achieves the best observed mean on all three tasks when each method uses its best observed learning rate within the tested grid. We hope this clarifies the statistical evidence and the scope of the additional comparison.

Reviewer_j1pPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FEED uses evaluation feedback to improve code search. It trains on valid programs and gives more weight to better solutions. It learns to prefer valid programs, and programs from earlier search rounds are reused for training. Experiments on several tasks measure solution quality and time to reach a target.

Critical strengths
  • The study use all valid programs for training. It also reuses programs from earlier rounds. This makes better use of limited search data.
  • The speed report includes the time updating the model. This makes the comparison more useful in practice.
  • The paper provides a detailed ablation analysis. The experiments test each loss and compare training with and without model updates.
  • The study tests FEED across several tasks and models. This shows its broad value.
Critical weaknesses
  • Preference pairs compare valid programs only with invalid ones. When candidates are valid but mediocre, this term gives no quality ranking, leaving improvement mainly to weighted self-training.
  • The bin-packing and routing heuristics are scored on the same fixed instances used during search. Appendix C.3 states there is no held-out evaluation, so improved scores do not establish a reusable heuristic for new instances.
  • Combining positive likelihood training with preference loss already exists, as Appendix A explains. FEED’s main addition is shared quality weights and reuse during search, making its method contribution useful but incremental.
  • Figures 2 and 3 appear to be AI-generated. More carefully crafted diagrams would better convey the technical content and strengthen confidence in the paper's quality and contributions.
Questions for authors

Please respond to the concerns raised in Critical Weaknesses.

Reviewer_RA8POpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FEED combines quality-weighted self-training with prompt-matched preference learning within contextual search. It selects valid programs from current evaluations and the historical learning pool. Both objectives share quality weights, while unpaired valid programs retain direct supervision. Model updates use a KL check and learning-rate backtracking. The authors compare FEED on six tasks under a time budget that includes generation, evaluation, and learning. The results show gains on some tasks, with additional model-size and model-family evaluations limited to 32-circle.

Critical strengths
  1. Evaluating discovery efficiency with all time costs included is practically useful. The authors include generation, evaluation, and learning in the time budget and assess both solution quality and time to threshold. This evaluation fits program discovery under limited resources. The time breakdown, threshold sensitivity analysis, and per-seed results also help assess the method's practical benefits.

  2. The experiments on some tasks support the value of joint learning and parameter updates. Results cover different types of program discovery tasks and include single-objective ablations and a fixed-model comparison. The results support that combining self-training and preference learning can provide gains that outweigh the additional learning cost on some tasks. Additional model-size and model-family comparisons provide evidence beyond the main model setting.

Critical weaknesses
  1. Improvements over the seed programs and the robustness of the results need further clarification. The search curves show progress, but evaluation scores for the seed programs and confidence intervals for the main results are missing. This limits the assessment of improvements within each task and differences between methods. Some maintenance details for the search archive and historical learning pool are also unclear, limiting reproducibility.

  2. The contributions of the main design choices have not been sufficiently isolated. The ablations support gains from joint learning on some tasks, but do not separate the contributions of quality weights, shared weights, historical valid programs, and direct supervision for unpaired valid programs. This limits the assessment of what the method adds.

Questions for authors
  1. Could the author provide the evaluation scores of the pre-supplied seed programs for each task and use the existing search curves to assess improvements over these starting points across tasks? Please also provide confidence intervals for RMST, mean best-so-far score, and differences between methods.

  2. With other settings unchanged, could the author compare uniform weights, quality weights applied only to self-training, and shared quality weights across both objectives to distinguish the contributions of quality weighting and weight sharing?

  3. Although parent selection uses a PUCT-inspired rule, could the author compare search with and without the exploration bonus to assess its effect?

  4. How do results change when training uses only valid programs from the current batch, or when self-training excludes valid programs without matched failures?

  5. Could the author clarify how the search archive is updated and how programs are removed, and whether the historical learning pool has a capacity limit? If these follow an former existing implementation, please provide corresponding sources.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.