- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a new method (XCONF) that leverages past experience to correct the confidence of the output. The method mainly contains two parts (RECALL and REFLECT). RECALL selects up to k past similar episodes from a task episode bank and measures the success rate. REFLECT takes the selected episodes as context and uses an LLM prompt to identify relevant failures and revise the confidence. The paper gives comprehensive experimental evaluation on 3 family models and 9 benchmarks.
Critical strengths
- The problem formulation and the defination of the episode are interesting.
- The paper shows diverse tasks and models in the experimental section.
- The ablation results are well-designed and strong; I like the part that tests the retrieval quality of RECALL.
Critical weaknesses
-
The system design is simple, and it heavily depends on the bank of episodes, and each episode requires a label of success. In practice, the bank seems to be difficult to build from scratch or extend to a large scale that covers long-tail tasks.
-
The design choice of combining the success rate in the retrieved episode and the reflection is not well justified; especially, the output of REFLECT is not well justified. Why is this reflection a reasonable design? For example, the recall is a sample of k past "similar" episodes. If the candidate pool is large and sucess/failr half half, if it happens to retrieve "all success" or "all failure", it will change the confidence(e) dramatically.
-
I noticed that the bank size is not reported for Table 2 and Table 3, and I am confused by the score reported in Figure 3; it seems that the max number reported in Figure 3 does not match Table 2. For example, the AppWorld result in Figure 3 is never larger than 0.9, but in Table 3, the result contains 0.91. Could you clarify the setting?
Questions for authors
- It seems that the bank depends on the model, since the episode contains the model's self-reflection and confidence states. And labeling the output is not easy. So this makes me think a similar setting: it seems that the SC@10 result is very close to XCONF, but XCONF needs to build a large bank. It seems that only label 10 SC outputs and using it to replace the RECALL is a reasonable baseline and good for cold start. Have you tried this?
- I notice that Figure 3 does not show a monotonic pattern with bank size increases. This links to my concerns that the larger number of past episodes will affect the RECALL and therefore change the results. Do you have an expectation of what would happen with a large number for each group? For example, in the setting that is given any input e, the number of similar candidates is not smaller than m.
4 comments
Q1: regarding labeling ten SC outputs as a cold-start baseline
Thank you for the suggestion; if we understand it correctly, it needs a label the estimator cannot have. You propose sampling ten SC outputs for the current task, labeling them, and using their success rate in place of Recall. Labeling them against ground truth requires the current task's answer, after which there is nothing left to estimate. Labeling them by majority vote makes the estimate SC@10 itself, which is in every table and which the self-labeling result of Section C shows cannot catch the confidently-wrong cases. And SC@10 is not close: 21 wins, 2 ties, and 1 loss against it with 3–8× lower ECE, and it does not apply to the agent domains. For cold start, Figure 11 is the relevant comparison: a bank of a hundred graded episodes already passes SC@10 where verbalized confidence is weak.
Q2: regarding non-monotonic Figure 3 and large banks
The curves need not be monotone, and a larger bank tightens the neighbourhood rather than destabilising the estimate. Each point in Figure 3 is a finite-sample estimate from fifty neighbours, so a wobble of a point or two at small banks is expected, and the trend over the full range is upward on every panel. In the regime you describe, where every input has at least m similar candidates, new episodes keep replacing neighbours with closer ones, so the neighbourhood becomes more homogeneous and the hit rate tracks the model's local success rate on that kind of task more closely; what remains is the sampling variance of a fifty-episode count, which does not grow with the bank. The evidence for this stability is empirical rather than theoretical: the k and σ ablation of Section I, and the flattening of the dense panels (MMLU-Pro at twelve thousand episodes), while BBEH and the agent domains are still climbing because they are not yet dense.
We hope this clears up your concerns; please let us know if anything remains unclear!
Thank you for the detailed answer. I find the explaination on cold-start and RECALL conflict. As you pointed out, the recall retrieves the local neighbourhood with . And it indicates that the size of the pool should be massive: suppose you have n tasks and each task has a neighbourhood much larger than 50 (in order to get high-quality 50 retrieval), the pool should be >> n*50. Consider the 100-200 episodes cold start setting: for most of task XCONF, retrieve 0 or few episode match the task (if task number > 200). So the RECALL part doesn't work. Another concern is the question I pointed out in Figure 3; I understand that monotonicity is not necessary, but given the small size from 500 to 1000 episodes in the science world, the increased episodes do decrease the AUROC a lot. I suspect this failure is very related to the RECALL design.
Thank you for following up on both points. We went back to the data for each, and the second one turned out to be an artifact of how we built Figure 3, which we should have caught earlier.
regarding cold start and the size of the neighbourhood
Recall does not need fifty episodes of the same task; at a hundred episodes it is mostly a calibrator of the stated confidence, and the gain from retrieval appears as coverage grows.
- What a neighbour has to share. The retrieval key weights the stated confidence most heavily (Section E), so a useful neighbour is a similar task met with similar confidence, and neighbourhoods of different tasks overlap. The n × 50 count assumes disjoint, same-task neighbourhoods.
- What a small bank gives (Table F1). We compare Recall with a calibrator that uses the stated confidence only, fit on the same bank. With 100 random episodes, about three per task type on ScienceWorld, Recall is already slightly ahead on ScienceWorld and by .05 AUROC on AppWorld, and level on SWE-bench; the margin widens as the bank grows.
- The case you describe, no matching episode at all. In the stored-order bank of 500 ScienceWorld episodes (see below), 89% of test queries belong to a task type with no episode in the bank. On those queries Recall scores .894 AUROC against .895 for the confidence-only calibrator. This is the fallback of Section 3.3: with nothing similar to retrieve, the estimate becomes the success rate at this stated confidence.
So you are right that retrieval contributes little at 100–200 episodes on some domains, and "already works" in our first reply overstated it. We will reword Section I to say that a small bank gives a calibrator and retrieval adds to it as episodes accumulate.
Table F1. Recall / confidence-only calibrator on the same bank, AUROC, random banks of B episodes (10 seeds), mean over the four models.
| B | ScienceWorld | AppWorld | SWE-bench Verified |
|---|---|---|---|
| 100 | .898 / .887 | .739 / .688 | .754 / .745 |
| 250 | .916 / .899 | .775 / .690 | .769 / .750 |
| full pool | .962 / .900 | .811 / .688 | .783 / .745 |
regarding the drop from 500 to 1,000 episodes on ScienceWorld
The drop comes from the order in which Figure 3 adds episodes, and our earlier explanation of it as sampling noise was incomplete.
- What happened. Figure 3 grows the bank in stored order, and ScienceWorld is stored grouped by task type. The first 500 episodes hold only the first few of thirty task types; the next 500 add whole new types with a very different success rate (for Claude Sonnet 4.6, 12% in episodes 250–500 against 54% in 500–1,000), while the test queries span all thirty. All four models share this prefix, which is why their curves dip together.
- With random banks the curve rises at every step (Table F2). Drawing the same number of episodes at random, over ten seeds, AUROC increases monotonically for each of the four models, with a standard deviation of at most .017 at 100 episodes and .006 from 500 on.
- Where your suspicion is right. A bank that covers few task types is worse than a random bank of the same size, and the reason is retrieval: queries from an uncovered type receive neighbours from other types. At the 1,000-episode point of the stored-order curve these queries sit about .03 below the confidence-only calibrator, and the gap closes once their types enter the bank. Coverage of task types, more than the raw episode count, is what a bank needs.
We will redraw Figure 3 with random subsets and a seed band, state the sampling in the caption, and keep the stored-order curve in the appendix as the low-coverage case.
Table F2. Recall AUROC on ScienceWorld as the bank grows, mean over the four models: stored order (Figure 3) vs random subsets (10 seeds).
| B | 100 | 250 | 500 | 1,000 | 2,000 | 4,000 | full pool |
|---|---|---|---|---|---|---|---|
| Stored order | .882 | .891 | .903 | .884 | .903 | .940 | .962 |
| Random | .898 | .916 | .930 | .943 | .952 | .958 | .962 |
Thank you for pressing on this figure; the paper is more accurate for it.
Thank you for the review, and for the positive words on the problem formulation and the retrieval-quality ablation. Since the ICLR committee encourages brevity, we keep this reply short; please let us know if anything remains unclear.
Thank you for this great question, which is also mentioned by other reviewers. Here we explain in detail:
Here we explain why reflect is needed and clarify a misunderstanding of recall retrieval:
Bank sizes are in Table 25, and the Figure 3 mismatch is a caption error on our side. Under five-fold rotation the bank behind each estimate is 80% of the per-column count in Table 25; we will add this to the captions of Tables 2 and 3. Figure 3 is a Recall-only replay with no Reflect calls, which is why its endpoints match Table 20's Recall-alone row (AppWorld: .810 for Claude, .888 for Gemini 3.5 Flash) rather than Table 3's blend (.911). The caption should have said so, and we will fix it.