- Rating
- 2
- Confidence
- 1
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies how feedback affects an LLM agent’s search and whether better proposals lead to a better final selected solution. The authors use a setting in which two LLMs search over fixed sets of classifiers trained before search for CIFAR-100 and GoEmotions. The experiments show that correct score labels improve average proposal quality, while improvements in final selected quality remain uncertain. Comparison format also changes exploration behavior depending on score visibility. The paper complements these experiments with theoretical results showing that proposal-quality sequences alone can be insufficient to determine final quality and characterizing when improving a proposal cannot worsen the selected outcome.
Critical strengths
-
Experimental design - The experiments keep candidate scores and the selection rule fixed while changing the feedback given to the LLM agents. Separate test samples evaluate proposal and final quality. This controlled design helps isolate how feedback changes search and whether its benefits survive selection.
-
Tracking candidate competition - The paper goes beyond average proposal quality and the frequency of trying new candidates. It tracks which proposals can challenge the starting winner and whether replacements help or harm performance. This provides an informative account of why better proposals may not produce a better final solution.
Critical weaknesses
-
Restricted experimental setting - The agents search over only 32 fixed classifiers with four proposal steps. About half the starting histories have no candidate that improves on the starting winner in both score and test quality. This leaves it unclear whether the findings extend to less restricted searches.
-
Limited confirmation of the main findings - Correct score labels improve average proposal quality in the original experiments, but the experiments provide limited confirmation that these gains exceed gains in final quality. Changes in search behavior are more clearly supported, while their effects on final quality remain uncertain. As I see it, this weakens the evidence for the paper’s broader conclusions.
Questions for authors
-
How did you train the 32 classifiers? Can you provide some details on their training and downstream performance? For example, how much do the 32 classifiers differ in their feedback scores and test performance? How strong are the four starting classifiers compared with the full set?
-
Why did you choose a search space of 32 classifiers and a budget of four proposals? Could these experimental choices limit the ability to improve the final solution?
-
When better proposals do not improve the final solution, is this mainly because they fail to replace the starting winner, or because the feedback score selects a candidate with lower test quality?
Thank you for asking how the strength of the starting classifiers affected selection. Scoring all 32 classifiers lets us distinguish three questions in each search. Was a better classifier available? Did the agent try it? If so, what kept it from being selected?
1. How were the classifiers trained, and how strong were the four starts?
The four starting classifiers ranged from strong to weak, but the one a new proposal had to beat was already among the strongest in the set. Within each block and target, the best of the four on validation had a median rank of 4.5 out of 32 on CIFAR-100 and 4 out of 32 on GoEmotions (Appendix A.2, p. 15, lines 777–779).
For CIFAR-100, we trained ResNet-18 classifiers from scratch for 12 epochs on the same images, varying image corruption and label smoothing. For GoEmotions, we trained one logistic classifier per emotion on the same comments, varying text features, regularization and class weighting. Both agents searched the same trained classifiers within a block (Appendix A.1–A.2, pp. 14–15, Table 3). For the table below, we averaged each configuration’s scores across blocks and targets before ranking it.
2. Did the search space and four-proposal budget limit improvement?
Yes. We chose a set of 32 classifiers so that we could evaluate even those the agent never tried. Four proposals gave both feedback conditions the same short budget. For about half the starting histories, none of the 32 classifiers beat that history’s starting validation winner on both validation and test. In those histories even trying every model could not improve the final test score under our selection rule (Section 2, p. 2, and Appendix B.1.4, p. 18).
Four rounds also limited discovery. In our added comparison of rounds four and eight within the same searches, more searches found a model better on both scores by round eight in all 12 classification settings. Yet when Terra trained new GoEmotions classifiers, extending search raised and lowered the selected model’s test score equally often. Extra rounds created opportunities that final selection did not consistently retain (Fig. 4a, p. 7, and Table 16, p. 38).
3. Why did a better proposal fail to become the final model?
Usually, the test-improving classifier failed to beat the starting model on validation. Here we count only original four-round searches that tried a classifier with a higher test score than the starting validation winner yet finished without a test improvement. In 125 of 132 such searches on CIFAR-100 and 174 of 181 on GoEmotions, every test-improving proposal tied or trailed the starting winner on validation. Those proposals would still have lost if all competing proposals were removed. Only in the remaining seven searches per task did a test-improving proposal beat the start on validation and lose to a different proposal (Table 15, p. 37, and Appendix B.1.4, p. 18).
For DeepSeek on CIFAR-100, the same finding held when the agent answered again from the same histories. Correct labels improved proposals more than the selected model in both batches. Mean selected accuracy fell slightly in the original searches because additional losses below the start outweighed additional gains (Table 2, lower panel, p. 5). It rose slightly with fresh responses. The original post hoc comparison supported a larger proposal gain in two of four settings (Table 1, p. 4). With fresh responses, only CIFAR-100/DeepSeek passed the prespecified approximate joint test, and none passed the bounded test (Table 4, p. 17).