- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents the confidence cliff as an interesting phenomenon in auto-research agents that their judgments/proposals tend to degrade over longer iterations. Then the paper proposes Rehearse as a mitigation skill that leverages multiple samples and efficient management of history to improve the model's judgment to select more effective attempts.
Critical strengths
- The paper presents a timely and interesting phenomenon called confidence cliff of LLM auto-research and potentially more broadly, related to long-horizon self-improvement (or recursive self-improvement);
- The paper conducts a controlled analysis illustrating confidence cliff, with ablation on components that can mitigate the cliff empirically;
- Experiments corroborate the end-to-end effectiveness of the proposed Rehearse skill;
- Experiment details are well documented and seem to be reproducible.
Critical weaknesses
- The paper does not provide an in-depth analysis into why confidence cliff happens, so other factors such as the model's long-context capabilities may confound this effect: Table 1 presents a spectrum of settings, from no history, full history, to summarized history, and Rehearsal's focused retrieved history, which has both the context length and context relevance changing at the same time while both stay potentially influential to the confidence cliff effect;
- Missing direct correlation between the severity of confidence cliff and performance; in particular, the current evidence in the paper only supports the claims that later proposals/judgments are less productive and reliable, but does not eliminate the possibility that the agent may use these sub-optimal options for exploration purpose, whose benefits are not immediate in downstream evaluations;
- The design of Rehearse, by itself, is not novel or surprising and the effectiveness largely leverages multiple samples plus a more efficient context management (similar idea was explored extensively in existing literature such as [1] and [2]).
[1] Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., & Yao, S. (2023). Reflexion: Language agents with verbal reinforcement learning. Advances in neural information processing systems, 36, 8634-8652.
[2] Cheng, Y., Zhu, X., Zhao, H., & Arora, S. (2026). Contextual Drag: How Errors in the Context Affect LLM Reasoning. arXiv preprint arXiv:2602.04288.
Questions for authors
- What is the mechanism of confidence cliff and why is it a necessary a bad phenomenon given its potential for more exploration?
- How is Rehearse different from a more efficient context management + more samples?
3 comments
I appreciate the authors' response. My remaining concerns are:
- Mechanism is still unclear, and the paper fails to deliver a tested explanation: the proposed explanation is a plausible reading but not tested. The two supporting observations (accuracy drops without history; history restores it) do not rule out a simpler alternative: the loop itself exhausts the obvious improvements, so successful late candidates are counterintuitive by construction. Under that reading, the cliff is a property of the candidate distribution at depth rather than a degradation of the judge.
- Direct control that separates length from relevance is still missing, confounding the interpretation of results: the outcome-label ablation is a useful fixed-length control, but it rests on roughly 46 verdicts, so the 83.5% vs. 77.5% gap is only a difference of about three; a more direct control would be to use randomly retrieved records from the same run at matched length.
- No direct connection to performance: the rebuttal argued conceptually potential exploration behaviors whose value is delayed, while this does not provide the direct evidence I asked for. There is no analysis relating the severity of the cliff to final performance or test of exploration value.
Since the core issues in my original review remain open, I am maintaining my score.
Thank you for the follow-up.
Mechanism. Your reading and ours describe the same thing: we do not claim that the judge model changes with depth. Section 6 says that early in a run general rules of thumb pick the better change, and that after several accepted changes whether one more change helps depends on what the run has already tried. That is the loop exhausting the obvious improvements, so the remaining successful candidates look counterintuitive from the code alone. The cliff is what this does to a judge inside a loop: on the pairs a loop actually faces at depth, its verdicts become more consistent across orders and less accurate, so the loop gets no signal that anything is wrong. What the paper tests is whether these late candidates are unpredictable, or only unpredictable without the run's own outcomes: on the same deep pairs, the same judge shown similar earlier attempts and their outcomes reaches 83.5%, against 56.9% without them. A loop cannot avoid reaching depth, so this is the decision it has to get right.
Length vs. relevance. The evidence is not the single 83.5% vs. 77.5% gap but the pattern across Table 2. The two conditions without selected record text sit at the two ends of prompt length and both give 70.8%: counts of similar successes and failures only (the shortest prompt), and all earlier records from the same store (the full dump, the longest). Every condition that keeps the candidate rationale and the similar records' text stays between 76.1% and 83.5%, including the single most similar record alone (top-1 only, 80.5%), a far shorter prompt than the dump. Accuracy follows whether the relevant records are in the prompt, not how long the prompt is. We will add the control you suggest, randomly retrieved records from the same run at matched length, to the final version.
Link to performance. The live runs give this link by intervention. Select and Rehearse share the proposer, the loop and the training-run budget, and differ only in whether the judge sees outcome memory, which the offline benchmark shows mitigates the cliff. On nanochat Rehearse ends ahead of Select on every seed (10.7% vs. 8.4% mean), and its additional gain comes later in the run, where the cliff is, while selection alone mainly speeds up early search (Section 5.2). In the same runs, after the third accepted change the memoryless selector's keep rate falls from 27.3% to 15.7%, while Rehearse's rises from 18.8% to 21.5%. If the memoryless arm's late wrong picks paid off as exploration, it would catch up by the end of the budget; instead the gap opens late in the run.
W1 (long context, length vs. relevance in Table 1). Long context cannot explain the cliff itself: the no-memory judge sees no history at any depth, and its accuracy still drops. The same happens in our live Select loop, whose judge also sees no history (Section 5.2). A judge that reads a growing history could add a second problem in long runs, the one you point to; our histories are short (about 2.8k input tokens per judgment in the online full-history arm), and that arm still ends below Rehearse on CIFAR-10 (2.45% vs. 2.85% mean gain).
Table 2 checks this. Keeping the retrieved records but removing only whether each one worked, a few words per record, leaves the length almost unchanged and drops deep-bucket accuracy from 83.5% to 77.5%. Replacing the records with counts of similar successes and failures, the shortest prompt, gives 70.8%. So the gain comes from what the records say, not from a shorter prompt.
W2 / Q1 (mechanism, exploration, link to performance). Our reading: after several accepted changes, the judge still sees the current code but not how the run got there, which changes were tried and how they turned out. Whether one more change helps depends on that path. Without it, the judge keeps applying the same general rule of thumb in both orders, so its verdicts stay consistent while accuracy drops. Two observations support this: the drop happens with a judge that reads no history at all, and giving it the history on the same deep pairs brings its accuracy back. The revision adds this explanation to Section 6.
On exploration: the judge is asked which candidate will improve the metric more, so a wrong verdict is simply a wrong prediction. An agent may choose to run an uncertain experiment to learn something, but a confident verdict does not mark a choice as exploration, and the loop cannot tell the two apart. That is why we treat the cliff as a problem. Whether late mistakes pay off later is what the online endpoint measures, since the best result after 100 runs already includes anything learned from failures. On nanochat the memoryless Select loop, which shows the score/keep-rate mismatch, ends at 8.4% mean BPB reduction, against 10.7% for Rehearse. The link to performance runs through the keep rate: after the third accepted change, where most of a run's training budget goes, the share of the memoryless selector's picks that are kept drops, while Rehearse's holds (Section 5.2).
W3 / Q2 (novelty: context management plus more samples?). The parts are simple on purpose, and we claim neither pre-run pairwise selection nor retrieval as new. Our contribution is the measurement: how well pre-run judgment works on pairs with known outcomes as the baseline changes, the cliff this reveals, and evidence that outcome memory mitigates it offline and helps in live loops. The finding is more than "a judge without information does worse". Before the third accepted change the same judge does about as well without run history; after it, far worse. And as it becomes less accurate, it becomes more consistent across presentation orders, not less. Both ingredients you name can be checked in our ablations. More samples alone do not help: proposing five candidates and running the first gives 6.9% on nanochat, against 7.1% for Vanilla; selection gives 8.4%, and selection with outcome memory 10.7%. Context alone does not explain it either: the same retrieved records without their outcome labels give 77.5%, against 83.5% with them (W1).
On [1] and [2]: Reflexion stores verbal reflections on a task's feedback and conditions the next trial of the same task on them, whereas Rehearse retrieves, for each pending candidate, the outcomes of other modifications already tried in the same run, and uses them before anything is executed. A Reflexion-style buffer is one of our baselines in Table 1 and reaches 74.1% in the deep bucket, against 83.5% for Rehearse. Contextual Drag shows failed attempts in context pulling a model's next solution toward similar errors; our judge produces no solution but predicts which pending change will work, and earlier failures, labeled as failures, are its evidence. The revision adds Contextual Drag and this comparison to Section 2.3.