acceptodds
Under review as a conference paper at ICLR 2027

Rehearse: Stepping Back from the Confidence Cliff in Autoresearch

Abstract

Improving a machine-learning system and maintaining the ability to identify its next improvement are distinct challenges for autoresearch. We isolate pre-execution judgment from proposal generation and execution, and examine how it changes as successful modifications accumulate. We construct a 366-pair benchmark from 39 paper-derived tasks in public AutoSOTA logs (Li et al., 2026; Tsinghua FIB Lab, 2026). Its primary 296 same-baseline pairs each match one modification that produced an accepted improvement with one that did not. With measured outcomes hidden, an LLM judge given candidate rationales but no prior-attempt history becomes more consistent across presentation orders but less accurate as successful changes accumulate. On the full benchmark, accuracy on agreed verdicts (selective accuracy) falls from 82.8% to 56.9%, while the fraction of pairs receiving an agreed verdict (strict-consensus coverage) rises from 76% to 85%. We call this mismatch the *confidence cliff*. **Rehearse** is a lightweight skill that proposes several ideas, retrieves similar past attempts and their outcomes to compare the candidates, runs the most promising, and records its outcome for future judgments. This focused outcome memory raises late selective accuracy to 83.5%. Across budgeted training runs over three loops, Rehearse improves the endpoint under the same training-run budget on nanochat, image classification, and time-series forecasting.

Primary area:
foundation or frontier models, including LLMs
Keywords:
autoresearch, LLM agents, pre-execution selection, focused outcome memory, LLM judges, selective prediction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pSj8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents the confidence cliff as an interesting phenomenon in auto-research agents that their judgments/proposals tend to degrade over longer iterations. Then the paper proposes Rehearse as a mitigation skill that leverages multiple samples and efficient management of history to improve the model's judgment to select more effective attempts.

Critical strengths
  1. The paper presents a timely and interesting phenomenon called confidence cliff of LLM auto-research and potentially more broadly, related to long-horizon self-improvement (or recursive self-improvement);
  2. The paper conducts a controlled analysis illustrating confidence cliff, with ablation on components that can mitigate the cliff empirically;
  3. Experiments corroborate the end-to-end effectiveness of the proposed Rehearse skill;
  4. Experiment details are well documented and seem to be reproducible.
Critical weaknesses
  1. The paper does not provide an in-depth analysis into why confidence cliff happens, so other factors such as the model's long-context capabilities may confound this effect: Table 1 presents a spectrum of settings, from no history, full history, to summarized history, and Rehearsal's focused retrieved history, which has both the context length and context relevance changing at the same time while both stay potentially influential to the confidence cliff effect;
  2. Missing direct correlation between the severity of confidence cliff and performance; in particular, the current evidence in the paper only supports the claims that later proposals/judgments are less productive and reliable, but does not eliminate the possibility that the agent may use these sub-optimal options for exploration purpose, whose benefits are not immediate in downstream evaluations;
  3. The design of Rehearse, by itself, is not novel or surprising and the effectiveness largely leverages multiple samples plus a more efficient context management (similar idea was explored extensively in existing literature such as [1] and [2]).

[1] Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., & Yao, S. (2023). Reflexion: Language agents with verbal reinforcement learning. Advances in neural information processing systems, 36, 8634-8652.

[2] Cheng, Y., Zhu, X., Zhao, H., & Arora, S. (2026). Contextual Drag: How Errors in the Context Affect LLM Reasoning. arXiv preprint arXiv:2602.04288.

Questions for authors
  1. What is the mechanism of confidence cliff and why is it a necessary a bad phenomenon given its potential for more exploration?
  2. How is Rehearse different from a more efficient context management + more samples?
3 comments

W1 (long context, length vs. relevance in Table 1). Long context cannot explain the cliff itself: the no-memory judge sees no history at any depth, and its accuracy still drops. The same happens in our live Select loop, whose judge also sees no history (Section 5.2). A judge that reads a growing history could add a second problem in long runs, the one you point to; our histories are short (about 2.8k input tokens per judgment in the online full-history arm), and that arm still ends below Rehearse on CIFAR-10 (2.45% vs. 2.85% mean gain).

Table 2 checks this. Keeping the retrieved records but removing only whether each one worked, a few words per record, leaves the length almost unchanged and drops deep-bucket accuracy from 83.5% to 77.5%. Replacing the records with counts of similar successes and failures, the shortest prompt, gives 70.8%. So the gain comes from what the records say, not from a shorter prompt.

W2 / Q1 (mechanism, exploration, link to performance). Our reading: after several accepted changes, the judge still sees the current code but not how the run got there, which changes were tried and how they turned out. Whether one more change helps depends on that path. Without it, the judge keeps applying the same general rule of thumb in both orders, so its verdicts stay consistent while accuracy drops. Two observations support this: the drop happens with a judge that reads no history at all, and giving it the history on the same deep pairs brings its accuracy back. The revision adds this explanation to Section 6.

On exploration: the judge is asked which candidate will improve the metric more, so a wrong verdict is simply a wrong prediction. An agent may choose to run an uncertain experiment to learn something, but a confident verdict does not mark a choice as exploration, and the loop cannot tell the two apart. That is why we treat the cliff as a problem. Whether late mistakes pay off later is what the online endpoint measures, since the best result after 100 runs already includes anything learned from failures. On nanochat the memoryless Select loop, which shows the score/keep-rate mismatch, ends at 8.4% mean BPB reduction, against 10.7% for Rehearse. The link to performance runs through the keep rate: after the third accepted change, where most of a run's training budget goes, the share of the memoryless selector's picks that are kept drops, while Rehearse's holds (Section 5.2).

W3 / Q2 (novelty: context management plus more samples?). The parts are simple on purpose, and we claim neither pre-run pairwise selection nor retrieval as new. Our contribution is the measurement: how well pre-run judgment works on pairs with known outcomes as the baseline changes, the cliff this reveals, and evidence that outcome memory mitigates it offline and helps in live loops. The finding is more than "a judge without information does worse". Before the third accepted change the same judge does about as well without run history; after it, far worse. And as it becomes less accurate, it becomes more consistent across presentation orders, not less. Both ingredients you name can be checked in our ablations. More samples alone do not help: proposing five candidates and running the first gives 6.9% on nanochat, against 7.1% for Vanilla; selection gives 8.4%, and selection with outcome memory 10.7%. Context alone does not explain it either: the same retrieved records without their outcome labels give 77.5%, against 83.5% with them (W1).

On [1] and [2]: Reflexion stores verbal reflections on a task's feedback and conditions the next trial of the same task on them, whereas Rehearse retrieves, for each pending candidate, the outcomes of other modifications already tried in the same run, and uses them before anything is executed. A Reflexion-style buffer is one of our baselines in Table 1 and reaches 74.1% in the deep bucket, against 83.5% for Rehearse. Contextual Drag shows failed attempts in context pulling a model's next solution toward similar errors; our judge produces no solution but predicts which pending change will work, and earlier failures, labeled as failures, are its evidence. The revision adds Contextual Drag and this comparison to Section 2.3.

I appreciate the authors' response. My remaining concerns are:

  • Mechanism is still unclear, and the paper fails to deliver a tested explanation: the proposed explanation is a plausible reading but not tested. The two supporting observations (accuracy drops without history; history restores it) do not rule out a simpler alternative: the loop itself exhausts the obvious improvements, so successful late candidates are counterintuitive by construction. Under that reading, the cliff is a property of the candidate distribution at depth rather than a degradation of the judge.
  • Direct control that separates length from relevance is still missing, confounding the interpretation of results: the outcome-label ablation is a useful fixed-length control, but it rests on roughly 46 verdicts, so the 83.5% vs. 77.5% gap is only a difference of about three; a more direct control would be to use randomly retrieved records from the same run at matched length.
  • No direct connection to performance: the rebuttal argued conceptually potential exploration behaviors whose value is delayed, while this does not provide the direct evidence I asked for. There is no analysis relating the severity of the cliff to final performance or test of exploration value.

Since the core issues in my original review remain open, I am maintaining my score.

Thank you for the follow-up.

Mechanism. Your reading and ours describe the same thing: we do not claim that the judge model changes with depth. Section 6 says that early in a run general rules of thumb pick the better change, and that after several accepted changes whether one more change helps depends on what the run has already tried. That is the loop exhausting the obvious improvements, so the remaining successful candidates look counterintuitive from the code alone. The cliff is what this does to a judge inside a loop: on the pairs a loop actually faces at depth, its verdicts become more consistent across orders and less accurate, so the loop gets no signal that anything is wrong. What the paper tests is whether these late candidates are unpredictable, or only unpredictable without the run's own outcomes: on the same deep pairs, the same judge shown similar earlier attempts and their outcomes reaches 83.5%, against 56.9% without them. A loop cannot avoid reaching depth, so this is the decision it has to get right.

Length vs. relevance. The evidence is not the single 83.5% vs. 77.5% gap but the pattern across Table 2. The two conditions without selected record text sit at the two ends of prompt length and both give 70.8%: counts of similar successes and failures only (the shortest prompt), and all earlier records from the same store (the full dump, the longest). Every condition that keeps the candidate rationale and the similar records' text stays between 76.1% and 83.5%, including the single most similar record alone (top-1 only, 80.5%), a far shorter prompt than the dump. Accuracy follows whether the relevant records are in the prompt, not how long the prompt is. We will add the control you suggest, randomly retrieved records from the same run at matched length, to the final version.

Link to performance. The live runs give this link by intervention. Select and Rehearse share the proposer, the loop and the training-run budget, and differ only in whether the judge sees outcome memory, which the offline benchmark shows mitigates the cliff. On nanochat Rehearse ends ahead of Select on every seed (10.7% vs. 8.4% mean), and its additional gain comes later in the run, where the cliff is, while selection alone mainly speeds up early search (Section 5.2). In the same runs, after the third accepted change the memoryless selector's keep rate falls from 27.3% to 15.7%, while Rehearse's rises from 18.8% to 21.5%. If the memoryless arm's late wrong picks paid off as exploration, it would catch up by the end of the budget; instead the gap opens late in the run.

Reviewer_9NcMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how agents choose the next change in an automated research loop. The authors build a test with 366 pairs from 39 tasks. They find that a judge becomes less accurate late in the loop, even when its answers agree across candidate orders. They call this the confidence cliff. Rehearse retrieves similar past changes and their results to help choose the next change.

Critical strengths

The paired test separates judgment from proposal generation. Each main pair shares a starting state and has measured results. I vote for this design because a final search score alone does not tell us whether the judge still makes good choices late in the loop.

Critical weaknesses
  • My main concern is the missing strong online memory baselines. In the offline deep group, a history summary gets 80.9% selective accuracy and Rehearse gets 83.5%. Over the full set, the scores are 81.9% and 82.3%. I do not find this gap strong enough on its own. The live tests do not compare summary memory or full history. ForeAgent also uses pairwise prediction before execution. I am not convinced that the proposed memory design is the best choice without these comparisons.
  • I am unsure how general the confidence cliff is. The deep group has 60 pairs, including 52 main type-1 pairs. Later changes may be harder or have less room to help. The same-baseline rule controls each pair, but not the difficulty across depth groups. The 0.5% minimum gain rule may also change which successful changes enter the test. The authors admit that depth is an observed variable. I think the broader explanation still needs stronger checks.
  • I do not think the paper has shown lower total research cost. Each five-candidate step needs 20 judge calls. The token check for the memory methods is useful, but it does not cover the full process. I want to count proposal calls, judging, retrieval, and actual training time. The current result supports better gain per budgeted training run. It leaves the total cost unclear.
Questions for authors
  • Do you have live results for summary memory or full-history memory under the same training budget? These comparisons would help me judge the value of focused retrieval.
  • What is the full cost of each method? Can you include model calls and actual GPU time, as well as the number of training runs?
1 comment

W1 / Q1 (online baselines). Both comparisons you asked for are done, on CIFAR-10. The two new arms differ only in what the judge reads: an LLM summary of the run's history (one per round, at most 512 words, shared by that round's judgments), or every earlier attempt with its outcome. Both new arms use the judge model of the original CIFAR-10 runs and five candidates per round, like Rehearse; all arms have the same 30-minute training budget and 100 attempts per seed, failed attempts included.

Table R1. Relative gain in accuracy over each run's own starting point after 100 attempts, mean ± s.d. over seeds.

ArmGain
Vanilla (one candidate, no comparison)2.10 ± 0.32%
Rehearse (retrieved outcomes)2.85 ± 0.38%
LLM summary2.62 ± 0.37%
All earlier attempts2.45 ± 0.47%

Rehearse ends highest. This matches offline: over all pairs Rehearse is only slightly ahead of the summary (82.3% vs. 81.9%), but in the deep bucket, the late pairs where the cliff appears, it leads by more (83.5% vs. 80.9%). Rehearse also uses the fewest tokens of the three memory arms (Table R2). Both tables are added to the revision (Appendix L, summarized in Section 5.3).

ForeAgent. It is the closest prior work, discussed in Section 2.1; the revision adds there what its predictor reads and how its accuracy is reported. We do not claim pre-run pairwise selection as new. ForeAgent's predictor reads the task, a verified data-analysis report and two complete solutions, and no outcomes of earlier attempts in the run. Its accuracy is pooled over a corpus of pairs, not tracked as the run progresses. What we add is the evaluation on a baseline that keeps changing, which is where the cliff shows up, and outcome memory at that step. Our Select arm is the same kind of pre-run pairwise selector, without ForeAgent's data report and confidence gate. On nanochat it beats Vanilla (mean relative BPB reduction 8.4% vs. 7.1%), and adding outcome memory gives 10.7%.

W2 (generality, difficulty across depth). Depth is observed, not controlled, as the paper says. Still, the cliff is not just that late pairs are harder: from the initial to the deep bucket the judge becomes more consistent across presentation orders while becoming less accurate (Figure 1). A judge that found these pairs harder would be expected to disagree with itself more often; it does the opposite. The memory comparison uses the same deep pairs, so difficulty is fixed there, and memory adds +26.6 pp (95% interval [5.1, 45.7]). The 0.5% threshold does not drive this either: on type-1 deep pairs where the successful candidate gained at least 1%, accuracy is 55.6% without memory and 82.1% with Rehearse (44 pairs).

The same thing happens online (Section 5.2), where the proposer reads the run's history and only the judge does not. In the nanochat Select runs the chosen candidate's score rises while its keep rate falls, in each run analyzed in Section 5.2; with Rehearse the keep rate holds. Most candidates executed in a live run come after the third accepted change, so this is where a run spends most of its budget.

W3 / Q2 (full cost of each method). The paper counts training runs (Section 7); Table R2 adds model calls and wall-clock time per round. Every round uses one of the 100 attempts, whether it trains or not.

Table R2. Per round. Time is the median, tokens the mean. Model calls are timed as direct API calls with the 20 judgments in parallel; each training run has the 30-minute budget. All rows are computed on the same CIFAR-10 rounds.

ArmProposalSummary20 judgmentsImplementationAll model callsTrainingRoundTokens
Vanilla41 s––6 s0.9 min30 min31.2 min28k
Rehearse41 s–29 s6 s1.4 min30 min31.7 min116k
LLM summary42 s28 s28 s6 s2.0 min30 min32.3 min123k
All earlier attempts41 s–30 s6 s1.4 min30 min31.7 min142k

Model calls take 3–6% of a round; training is most of it, so the ranking by gain per training run also holds per hour of wall clock. Rehearse uses slightly fewer tokens than the summary arm, which writes a summary every round, and fewer than the full history, whose prompts grow over the run (87k per round early, 192k late).

Reviewer_nndvOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Rehearse, a pipeline to fix issues in autoresearch where agents get worse and worse at selecting which code change to run as they receive success in the research loop. They built a benchmark of 366 pairs of code changes from the AutoSOTA logs. They find that in most pairs of code, some changes lead to success and others to failures even though the starting point is the same. They fin show that using an LLM judge with no memory picks the right change 82.8% of the time early in the conversation, but only 56.9% of the time after three or more changes have been accepted by the system. They also find that the judge agrees with itself (confidence cliff) and propose to fix it through Reherse. Reherse finds similar past attempts and runs a pairwise tournament, raising the accuracy to . This brings to 83.5%. Their method beats the standard research loop on on nanochat, CIFAR-10 and ETTh1.

Critical strengths
  • They motivate the problem well and have built the benchmark carefully
  • They perform a common-verdict check (82.5% vs 63.4%) showing that improvements are not from issuing fewer verdicts.
  • They run strong ablations i.e. nanochat. They find that proposing more ideas and running the first one does nothing (6.9% vs 7.1%), while selection (8.4%) and memory (10.7%) help. The finding that adding failure reasons hurts accuracy is surprising and useful.
  • The method does not require additional fine-tuning
Critical weaknesses
  • The paper is very hard to read.... Many sentences are dense and the gap identified / contribution was not incredibly clear
  • The paper does not report error bars, and they only measure confidence cliff on 60 pairs
  • At ~77% coverage, Rehearse and the LLM summary each give about 46 verdicts, so the gap between them (83.5% vs 80.9%) may not be statistically significant
  • It is unclear which judge models are used, and what the errors bars and results were for each one. How are the LLM judges validated?
  • The paper says the judge gets worse later in the loop but never explains why?
  • Is there an error in Table 2? I see that two different ablations show exactly the same numbers (80% and 70.8%), which also match a row in Table 1. Could you clarify?
  • A change only counts as “worked” if it improved by at least 0.5% (from a single run). We see that most gains are around 0.3% later in the loop.... but could this just be from noise?
Questions for authors
  1. Which three judge models are used, and what are the per-model results?
  2. Does memory provide benefits when very similar records are excluded?
  3. What are the token and wall-clock costs in the live loops?
2 comments

W1 (readability, gap, contribution). Most autoresearch loops run whatever the agent proposes, so their efficiency depends on a choice that is rarely measured: which change deserves the next training run, when the baseline keeps changing. We measure that choice on candidate pairs with known outcomes. As accepted changes accumulate, the judge becomes more consistent across the two presentation orders but less accurate; we call this the confidence cliff. Showing it the outcomes of similar earlier attempts recovers much of the lost accuracy, and it also helps in live loops. The revised introduction follows this order and defines coverage and selective accuracy where they first appear.

W2 (error bars, 60 deep pairs). The curves in Figure 3 carry ±1 s.d. bands over five runs; Figure 1 has none, so here are intervals for its main comparison. In the deep bucket Rehearse is +26.6 pp above no memory (95% interval [5.1, 45.7]), and +28.0 pp on the 52 type-1 pairs alone ([4.5, 53.5]). The intervals are computed over tasks, the more conservative choice. These intervals are now in Appendix K.

W3 (83.5% vs. 80.9%). About 46 verdicts per condition are too few to separate a gap this small, so we ran the online comparison on CIFAR-10, with the same budget and 100 attempts per seed. Rehearse again ends highest: a mean gain of 2.85 ± 0.38%, against 2.62 ± 0.37% for an LLM summary of the history and 2.45 ± 0.47% for the full history. It also uses no more tokens, about 116k per round against 123k and 142k.

W4 / Q1 (judges, per-model results, validation). No model produces the labels. Every verdict is checked against the measured results of the two candidates, which were both actually trained, and the judge never sees those results. In type-1 pairs both candidates start from the same code. Each pair is judged twice with the order swapped and counts only if both answers agree. Coverage is the share of pairs that get a verdict; selective accuracy is the share of those verdicts that are right.

Both curves in Figure 1 are means over three judges, hy3-preview, glm-5.1 and deepseek-v4. Per model (also in Appendix K):

JudgeCoverage, all pairsSelective accuracy, all pairsSelective accuracy, deep bucket
hy3-preview82.2%83.7%84.4%
glm-5.182.0%83.0%84.8%
deepseek-v483.3%80.3%81.3%

W5 (why the judge gets worse). Early in a run, general rules of thumb work: a larger learning rate or more regularization often helps. After several changes are accepted, the judge still sees the current code, but not how the run got there: which changes were already tried, and which worked or failed. Whether one more change helps now depends on that path. Without it the judge falls back on the same rule of thumb whichever order the pair is shown in, so its verdicts stay consistent while its accuracy drops. That is the pattern in Figure 1. This judge reads no history at any depth, so long context is not the cause. Memory supplies exactly that path, and it matters mainly late: before the third accepted change memory moves accuracy by only a few points either way, and after it by 26.6 points. The revision adds this explanation to Section 6.

W6 (identical numbers in Table 2). It is a coincidence, not a copy. Two separate runs happen to give the same totals (48 verdicts on the 60 deep pairs, 34 of them right), but they disagree on 30 of the 60 pairs. The full-history row of Table 2 is the same condition as the Table 1 row with these numbers, repeated there for comparison. The caption of Table 2 now says both.

W7 (the 0.5% threshold and the ~0.3% late gains). The two numbers are about different things. 0.5% is the minimum gain for a candidate to count as "worked" in our benchmark (the smallest one admitted gained 0.517%). 0.3% is the average gain of helpful changes late in the full logs. Labels do come from single runs, so we checked on the type-1 deep pairs whether borderline pairs drive the cliff:

SubsetPairsNo memoryRehearse
Successful candidate gained ≥ 1%4455.6%82.1%
Failed candidate got strictly worse4356.8%83.5%

Neither subset differs much from the whole deep bucket (56.9% without memory, 83.5% with Rehearse), so borderline pairs are not behind the effect.

Q2 (memory without very similar records). We have not tried an upper similarity cutoff, but our online runs show that copying the outcome of a very similar record would not work. In the CIFAR-10 runs, when the most similar earlier attempt (cosine ≥ 0.7) had worked, the new attempt still failed 86% of the time (56 of 65). Section 5.2 gives a case: lowering weight decay failed while raising it worked. The answer also cannot leak: the two candidates being judged are never retrievable, and only attempts run before them can be retrieved.

Q3 (token and wall-clock cost). A round takes 31–32 minutes: the 30-minute training run plus about 1 to 2 minutes of model calls (with Rehearse, five candidates and 20 judge calls). Per arm, on the same CIFAR-10 rounds:

ArmModel callsRoundTokens
Vanilla0.9 min31.2 min28k
Rehearse1.4 min31.7 min116k
LLM summary2.0 min32.3 min123k
Full history1.4 min31.7 min142k

Model calls are 3–6% of a round, so the ranking by gain per training run also holds per hour of wall clock. This table is in Appendix L.

Comments on the paper

1 comment
AuthorsOpenReview ↗

To all reviewers

Thank you for the reviews. All three of you asked how Rehearse compares with simpler memory, and two of you asked what it costs beyond training. We ran new experiments for both.

On CIFAR-10 we added two live arms whose judge reads either an LLM summary of the run's history or the full history, 100 attempts per seed. Rehearse ends highest: a mean gain of 2.85 ± 0.38%, against 2.62 ± 0.37% for the summary and 2.45 ± 0.47% for the full history. This matches the offline result, where Rehearse is also ahead of the summary in the deep bucket (83.5% vs. 80.9%). It is also the cheapest of the three memory arms: a Rehearse round uses about 116k tokens, against 123k for the summary and 142k for the full history, and it needs no summary call. Each reply below gives the numbers for the questions it answers.

Changes in the revised PDF are marked in blue:

  • online memory baselines and the full cost per round (Appendix L, summarized in Section 5.3);
  • why late judgments fail (Section 6);
  • task-level intervals and per-judge results (Appendix K);
  • ForeAgent and Contextual Drag (Sections 2.1 and 2.3);
  • a rewritten introduction, and the three judges named in the caption of Figure 1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.