acceptodds
Under review as a conference paper at ICLR 2027

HOW FEEDBACK SHAPES SEARCH: Candidate Competition in LLM Agents

Abstract

Large language model (LLM) agents use feedback to guide search, but better proposals need not yield a better final solution. We study two LLMs searching fixed sets of image and text classifiers trained before search. With evaluation and selection fixed, showing the correct condition label for each score improves average proposal quality for the requested condition in all four original model–task settings. Paired post hoc comparisons support larger effects on proposal quality than on final quality in two settings, with limited confirmation from new responses. In text classification, the same comparison format encourages one model to revisit tried configurations or explore untried ones depending on score visibility. We construct searches with identically distributed complete proposal-quality sequences but different final outcomes under one selection rule, and characterize when improving a final proposal cannot reduce selected quality. Evaluating feedback therefore requires tracking candidate competition and improvements preserved through selection.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, feedback, search, candidate selection, agent behavior

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EZYUOpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies how feedback affects an LLM agent’s search and whether better proposals lead to a better final selected solution. The authors use a setting in which two LLMs search over fixed sets of classifiers trained before search for CIFAR-100 and GoEmotions. The experiments show that correct score labels improve average proposal quality, while improvements in final selected quality remain uncertain. Comparison format also changes exploration behavior depending on score visibility. The paper complements these experiments with theoretical results showing that proposal-quality sequences alone can be insufficient to determine final quality and characterizing when improving a proposal cannot worsen the selected outcome.

Critical strengths
  1. Experimental design - The experiments keep candidate scores and the selection rule fixed while changing the feedback given to the LLM agents. Separate test samples evaluate proposal and final quality. This controlled design helps isolate how feedback changes search and whether its benefits survive selection.

  2. Tracking candidate competition - The paper goes beyond average proposal quality and the frequency of trying new candidates. It tracks which proposals can challenge the starting winner and whether replacements help or harm performance. This provides an informative account of why better proposals may not produce a better final solution.

Critical weaknesses
  1. Restricted experimental setting - The agents search over only 32 fixed classifiers with four proposal steps. About half the starting histories have no candidate that improves on the starting winner in both score and test quality. This leaves it unclear whether the findings extend to less restricted searches.

  2. Limited confirmation of the main findings - Correct score labels improve average proposal quality in the original experiments, but the experiments provide limited confirmation that these gains exceed gains in final quality. Changes in search behavior are more clearly supported, while their effects on final quality remain uncertain. As I see it, this weakens the evidence for the paper’s broader conclusions.

Questions for authors
  1. How did you train the 32 classifiers? Can you provide some details on their training and downstream performance? For example, how much do the 32 classifiers differ in their feedback scores and test performance? How strong are the four starting classifiers compared with the full set?

  2. Why did you choose a search space of 32 classifiers and a budget of four proposals? Could these experimental choices limit the ability to improve the final solution?

  3. When better proposals do not improve the final solution, is this mainly because they fail to replace the starting winner, or because the feedback score selects a candidate with lower test quality?

1 comment

Thank you for asking how the strength of the starting classifiers affected selection. Scoring all 32 classifiers lets us distinguish three questions in each search. Was a better classifier available? Did the agent try it? If so, what kept it from being selected?

1. How were the classifiers trained, and how strong were the four starts?

The four starting classifiers ranged from strong to weak, but the one a new proposal had to beat was already among the strongest in the set. Within each block and target, the best of the four on validation had a median rank of 4.5 out of 32 on CIFAR-100 and 4 out of 32 on GoEmotions (Appendix A.2, p. 15, lines 777–779).

For CIFAR-100, we trained ResNet-18 classifiers from scratch for 12 epochs on the same images, varying image corruption and label smoothing. For GoEmotions, we trained one logistic classifier per emotion on the same comments, varying text features, regularization and class weighting. Both agents searched the same trained classifiers within a block (Appendix A.1–A.2, pp. 14–15, Table 3). For the table below, we averaged each configuration’s scores across blocks and targets before ranking it.

TaskRange of mean validation scoresRange of mean test scoresFour starting classifiers’ ranks by mean test score (1 = best)
CIFAR-100, accuracy (%)49.75–59.4250.17–59.541, tied 11, 26, 32
GoEmotions, micro-F1 (0–100)10.78–33.7710.40–33.176, 8, 20, 32

2. Did the search space and four-proposal budget limit improvement?

Yes. We chose a set of 32 classifiers so that we could evaluate even those the agent never tried. Four proposals gave both feedback conditions the same short budget. For about half the starting histories, none of the 32 classifiers beat that history’s starting validation winner on both validation and test. In those histories even trying every model could not improve the final test score under our selection rule (Section 2, p. 2, and Appendix B.1.4, p. 18).

Four rounds also limited discovery. In our added comparison of rounds four and eight within the same searches, more searches found a model better on both scores by round eight in all 12 classification settings. Yet when Terra trained new GoEmotions classifiers, extending search raised and lowered the selected model’s test score equally often. Extra rounds created opportunities that final selection did not consistently retain (Fig. 4a, p. 7, and Table 16, p. 38).

3. Why did a better proposal fail to become the final model?

Usually, the test-improving classifier failed to beat the starting model on validation. Here we count only original four-round searches that tried a classifier with a higher test score than the starting validation winner yet finished without a test improvement. In 125 of 132 such searches on CIFAR-100 and 174 of 181 on GoEmotions, every test-improving proposal tied or trailed the starting winner on validation. Those proposals would still have lost if all competing proposals were removed. Only in the remaining seven searches per task did a test-improving proposal beat the start on validation and lose to a different proposal (Table 15, p. 37, and Appendix B.1.4, p. 18).

For DeepSeek on CIFAR-100, the same finding held when the agent answered again from the same histories. Correct labels improved proposals more than the selected model in both batches. Mean selected accuracy fell slightly in the original searches because additional losses below the start outweighed additional gains (Table 2, lower panel, p. 5). It rose slightly with fresh responses. The original post hoc comparison supported a larger proposal gain in two of four settings (Table 1, p. 4). With fresh responses, only CIFAR-100/DeepSeek passed the prespecified approximate joint test, and none passed the bounded test (Table 4, p. 17).

Reviewer_e3HVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies a how LLMs search in response to different feedback formats in a toy classifier search setting. The authors use clean simple experiments to show that the best strategy for agents (extrapolating from the toy setup) is one that incorporates feedback wrt to the best solution seen so far instead of just improving the current candidate. The paper is quite dense but the experiments are good. My main concern is how useable these empirical results given such a simple task(s) and only 2 LLMs tested.

I did not use AI.

Critical strengths
  • Clean experiment design. Makes it easy to identify the effects of different feedback proposals
  • Useful direction. In the era of agents, many users and researchers take for granted how to maximize these tools and this study examines a key component (how to give feedback) in these agentic loops.
Critical weaknesses
  • Very simplistic toy setting experiments. Task is selecting best config from 32 classifiers, e.g. for cifar classification. This task is far too simplistic to takeaway conclusions for real-world agentic tasks.
  • Unclear if the conclusion on proposal-versus-final gains is correct. It seems the proposal gain gap w.r.t to final selection is only significant in 2 experiments.
  • Only tested on 2 LLMs. Unclear if findings hold for more models. Or even if results hold when varying reasoning (e.g. the display formatting may vary if the model uses more or less thinking potentially).
Questions for authors

See weaknesses.

1 comment

Thank you for encouraging us to go beyond two agents choosing from 32 pretrained classifiers. Your suggestion led us to add Qwen3.8-27B as a third agent and compare two ways of searching. From the same starting classifiers and with the same eight attempts, each agent could either choose pretrained models or propose new ones to train, including image architectures on CIFAR-100 and classifier code on GoEmotions (Section 2, p. 3, and Appendix E.1, pp. 31–32).

1. Does the finding extend to agents that train their own models?

Qwen provides a clear example beyond the original set. On CIFAR-100, even counting only classifiers it successfully trained, correctly identifying which validation score belonged to the requested target improved mean proposal accuracy by 1.55 percentage points more than it improved the accuracy of Qwen’s final choice (nominal 95% CI [0.71, 2.39], Table 13, p. 36). For DeepSeek and Terra, switching from pretrained classifiers to training new ones narrowed the gap (Fig. 3a,b, p. 6, and Appendix F.2, p. 35). The original set affected how large the gap was. Qwen shows why it is still worth checking which gains reach the selected model when the agent designs its own classifiers.

The GoEmotions training runs did not establish a comparable gap, nor did WMT24 translation fine-tuning when we averaged over both directions (Table 12, p. 35). BIRD text-to-SQL tested a different change, adding detail to the feedback. There too, some searches tried a model that beat the start on both validation and test, yet selected one that did not improve on the start on test (Appendix F.4, p. 37).

2. What do the original comparisons establish?

You are right that two of the four original settings supported a larger proposal gain in the post hoc comparison, DeepSeek on CIFAR-100 and Terra on GoEmotions (Table 1, p. 4). With fresh responses, only the former passed the prespecified approximate joint test, and none passed the bounded test (Table 4, p. 17).

The four settings agree on where most of the rise in mean proposal quality came from. Correct labels reduced the share of proposals that could not beat the start on validation, and those still below or tied with the start came closer to it on test. Closing that deficit raises the proposal average even when the model cannot be selected (Section 3, p. 4, and Table 5, p. 18). We also asked what happened to gains that could have been selected. In the additional eight-round CIFAR-100 searches over pretrained classifiers, each of the three agents left more of the available test gain unselected under correct labels than under swapped labels. This comparison counts improvement over the start only from proposals that beat it on validation (Table 15, p. 37).

3. Does reasoning effort change how Terra responds to the display?

The comparison did not establish that reasoning effort changes Terra’s response to sorting. We tested both effort settings on the same GoEmotions histories. At low effort, sorting made revisits more frequent when only some scores were visible. At high effort, sorting had little effect on the next choice with either partial or full scores. Appendix E.4 (p. 34) describes the comparison and Appendix F.6 (p. 39) reports the results.

Reviewer_nGLJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how feedback affects LLM agents’ search behavior and whether better candidates lead to better final results. The authors use two LLMs to choose from fixed sets of image and text classifiers trained before the search. They change which condition each score is labeled as belonging to, how many scores the agents can see, and how the scores are presented for comparison. The evaluation method and final selection rule stay the same. Correct score labels improve average candidate quality in all four original model–task settings. However, only two settings support a larger improvement in candidate quality than in final quality, and rerunning the experiments with fresh LLM responses provides limited support for this finding. The paper also gives a theoretical example showing that searches with the same distribution of candidate-quality sequences can still produce different final results under the same selection rule.

Critical strengths
  1. The research question is interesting and relevant to LLM agents that learn from feedback during search. The paper makes a useful distinction between proposing better candidates and actually returning a better final solution.
  2. The experiments are carefully controlled. The classifiers are trained beforehand, and their scores and the final selection rule stay fixed. This makes it easier to see how feedback changes the agents’ choices.
  3. The effect of score presentation is interesting. For one model, the same format encourages it to revisit old configurations when only some scores are visible, but try new configurations when all scores are visible. This shows that the usefulness of a presentation format depends on what information the agent already has.
Critical weaknesses
  1. The main finding, that feedback improves candidate quality more than final quality, does not receive consistent support in the follow-up experiments. The original comparison was added after the initial analysis. With fresh responses, only one of the four settings passes the approximate joint test, and none passes the bounded-analysis test. DeepSeek’s reasoning setting also changes in the follow-up, which makes the results harder to compare.
  2. The search space is limited to 32 candidates, with only a few search rounds. In about half of the starting situations, no candidate in the pool can beat the initial winner on both the selection score and hidden test performance. This leaves little room to improve the final result. Table 9 examines this issue, but it is still unclear whether the same findings would hold when there is more room for improvement.1. The research question is interesting and relevant to LLM agents that learn from feedback during search. The paper makes a useful distinction between proposing better candidates and actually returning a better final solution.
  3. The link between the theory and the experiments could be clearer. In the recorded GoEmotions searches, cases where the exact candidate-quality sequence leaves the final result uncertain are rare. The approximate-matching result broadens the theory, but its relevance to the observed searches needs more explanation.
Questions for authors
  1. Building on Table 9, if we focus on starting situations where a better candidate is available, how strong is the evidence that candidate quality improves more than final quality?
  2. Why was DeepSeek’s reasoning setting changed in the fresh-response experiments? Are results using the original setting available?
  3. Could the authors give an example from the recorded searches showing how the approximate-matching result helps explain the observed behavior?
1 comment

Thank you for suggesting a closer look at Table 9 of the original submission (p. 20). That table showed how much each group of starts contributed to the overall gap. Your question led us to estimate the gap within the group where a better classifier was available.

1. Does the gap persist when a better model is available?

Even with room to improve, correct labels benefited the models DeepSeek tried on CIFAR-100 more than the model it ultimately selected. We restricted the original searches to starting histories where at least one of the 32 classifiers beat the starting validation winner on both validation and test. Across the 26 contributing blocks, the gain in mean proposal test accuracy exceeded the gain in selected-model test accuracy by 1.44 percentage points (nominal 95% CI [0.29, 2.59]). In the other three settings the intervals included zero (Appendix B.1.4, p. 18, lines 946–956).

The fresh-response experiment in the original submission addressed a separate question. It used all starting histories to test whether the result recurred when the agents answered again. Two of the four settings supported a larger proposal gain in the initial post hoc comparison (Table 1, p. 4). With fresh responses, only DeepSeek on CIFAR-100 passed the prespecified approximate test requiring a positive proposal gain that also exceeded the final-model gain. None passed the bounded sensitivity test (Table 4, p. 17).

2. Which reasoning setting did DeepSeek use?

Both batches used the high setting. The original client requested it explicitly (Table 1, p. 4). After we switched service interfaces, the fresh-response client used the provider’s high default with an 8,192-token output cap (Appendix C.2, p. 27, lines 1427–1431). We did not collect a fresh batch through the original interface. The logs do not record the providers’ internal reasoning budgets.

3. What does approximate matching explain in a recorded search?

Two DeepSeek searches on GoEmotions show what the proposal test scores leave out. The searches had the same starting models and first two proposals. Their third proposals scored almost identically on test, but only one beat the starting winner on validation. That proposal displaced a model with a higher test score. The other search retained that model. The proposal test scores differed by at most 0.12 micro-F1 points at each step, yet the final models differed by 5.01 points. This was the only one of 24 approximately matched pairs with a final gap of at least one point on the original test comments. On a second test sample this pair no longer met the matching criterion. The example illustrates how validation can change the winner even when proposal test scores are nearly identical (Appendix D.3–D.4, p. 29).

Comments on the paper

1 comment

Thank you all for pointing out where the original manuscript was hard to follow. We rewrote the introduction (p. 1) around two questions. Does feedback help the agent try better models? How much of that improvement reaches the model it selects? Ji et al. compare selection rules on fixed pools of answers (p. 1). We held the rule and every classifier’s scores fixed to isolate how changing the feedback altered which models competed for selection. Figure 2 (p. 3) shows recorded examples.

Section 3 shows why that distinction changes how one would judge a feedback design. On CIFAR-100, correct labels raised the mean test accuracy of the classifiers DeepSeek tried. They also doubled the number of original searches whose final classifier beat the start on test. By either measure, correct labels looked preferable. Yet more searches also finished below the start, and the additional losses outweighed the additional gains (Tables 1–2, pp. 4–5). Mean final accuracy fell slightly in that batch and rose slightly when the agent answered again (Table 4, p. 17). In both batches the proposals gained more. Figure 5 (p. 8) brings together the corresponding search and selection outcomes for the score-display experiment.

Theorem 1 explains why recording every proposal’s test score still leaves something out. Both searches in the construction always try a model with the best test score. They have the same distribution of test scores in order, yet which models appear together changes which one wins on validation (Section 6.2, pp. 9–10). This motivates a comparison researchers can make on recorded searches. Each time a new model takes the validation lead, compare its held-out performance with that of the model it displaced.

The reviews also prompted new experiments. We added a third agent and compared choosing existing classifiers with proposing new ones to train, holding the starting models and search budget fixed. Figure 3 (p. 6) shows this eight-round comparison on CIFAR-100. Table 1 reports the original four-round searches.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.