- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies why a retrieved, verified-successful trajectory may be relevant to an LLM agent’s task yet fail to help it. It proposes FAMA, a selector that reads the agent’s failed rollout and chooses a memory for one retry. Across code generation, embodied interaction, and text-to-SQL, the fine-tuned selector recovers 8–11 percentage points more initially failed tasks than top-1 similarity retrieval.
Critical strengths
-
The paper tests retrieved memories individually and compares FAMA with a selector that sees the same task and candidates but not the failure. This makes the central finding more convincing than an improvement over similarity retrieval alone.
-
Fine-tuned FAMA recovers 8-11% more previously failed tasks than top-1 retrieval across code generation, embodied interaction, and text-to-SQL, demonstrating benefits that generalize beyond a single benchmark.
Critical weaknesses
-
The cross-encoder does not see the failed rollout, and on KodCode FAMA can choose separate memories for two agents while some baselines must use one shared memory. This leaves uncertainty about how much of the reported gain comes from FAMA’s proposed method rather than extra information or selection flexibility.
-
The experiments use verified successful, same-distribution memories and require a reliable signal that the first attempt failed. Without results for noisy banks or uncertain outcomes, it is unclear whether the gains extend to less controlled agent settings.
Questions for authors
-
On KodCode, how much of FAMA’s gain remains if it must select one shared memory for both agents, matching the top-1 and cross-encoder baselines?
-
How does FAMA compare with a failure-aware reranker trained or prompted using the same failed rollout and the same ten candidates? Can you separate the value of seeing the rollout from the value of the proposed diagnosis-and-selection procedure?
-
What are the paired confidence intervals for the main recovery differences, and what are the added selector cost and end-to-end cost per recovered task?
2 comments
Thanks for the very detailed response and the further experiments. The new robustness, stability, geometry and numerical bound results go a long way to addressing my primary concerns and the controlled 2D example does provide some useful evidence that the geometry sensitive bound can be of value. The rebuttal does provide a lot of empirical backing, though the remaining looseness of the practically relevant bounds means that a large score increase is not really justifiable. I'd keep my rating.
Dear reviewer,
Thank you for the precise questions. Please also see our General Response (top-level comment) for a table of all new experiments and the cost analysis. New results are recovery rates (% of memory-free failures solved on retry) on KodCode / ALFWorld / Spider, same executor and same ten candidates as Table 1.
Failure-aware reranker; rollout vs. procedure. We added the failed trajectory to the cross-encoder query (same off-the-shelf bge-reranker as in Table 1, not retrained): 39.3 / 31.5 / 36.5 vs. 39.6 / 39.0 / 36.2 without it. No gain: a reranker that sees the rollout but was not trained to interpret it does not benefit from it. To separate seeing the rollout from the diagnosis procedure, we fine-tuned the same model on the same instances to output the choice directly, with no diagnosis text: 50.8 / 45.1 / 37.0, vs. FAMA 49.7 / 48.4 / 41.0. So seeing the rollout is worth +8.9 / +3.8 / +0.5 over the selector w/o failure, and the diagnosis adds −1.1 / +3.3 / +4.0 on top. On KodCode the rollout alone suffices; on ALFWorld and Spider the procedure carries a real part of the gain. This matches Analysis I-a: on KodCode the failure's surface already carries the signal, on the other two it does not.
Cost. Please see the cost table in the General Response. Per retried task, FAMA costs 15–37% more than Top-1 (the selector call takes 1.2–1.8 s, less than one executor retry at 4–11 s). Per successful recovery, FAMA is on par with Top-1 and 2–6× cheaper than Reflexion, because it recovers a much larger share of the tasks it retries.
Uncertain outcomes. FAMA is triggered by a failure signal, which in our settings comes from the environment. How FAMA behaves when that signal is noisy or self-judged is a separate question from the selector itself, and we leave it to future work.
We thank you again for the careful review, and we look forward to your reply.