acceptodds
Withdrawn from ICLR 2027

CAVRA: Causal Adaptive Verified Remasking for Self-Evolving Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language model agents are increasingly deployed as long-horizon controllers, yet they are almost always served with frozen weights, so any lasting improvement must live in the external textual policy—skills, plans, and memory—rather than in the parameters. Turning a failed trajectory into a policy that transfers remains obstructed by two protocol gaps: a teacher that locates the defect either consults privileged information that vanishes at deployment or else rewrites the entire episode, and a skill written from the remasked instance itself contaminates later tasks. We therefore propose CAVRA, a remask–recover framework in which the same frozen model is both the student that acts and the teacher that localizes failure. Causal remask attribution spends a bounded mask budget on the high-credit control span, reading the world's own stall signature rather than a privileged judge. Local recover then inpaints only those spans and scores each fill with a progress-aligned probe confined to the error region, so the official first-pass verdict is never overwritten. Utility-latched precipitation crystallizes a skill from remask kinds and failure reasons—or from already-corrected local fixes—after instance facts have been stripped, and family-conditioned routing injects only the matching skeleton. Extensive experiments, including a matched-backbone comparison against ten baselines, module ablations, and adaptation-cost measurements, show that CAVRA improves interactive control substantially, leading the strongest baseline by 40.6% relative success on ALFWorld.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
self-evolution, large language models, agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LY5oOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies self-evolution of LLM agents when the backbone is frozen and only a textual skill store can change. When a task fails, the same model acts as a teacher and marks at most three verbatim spans that caused the failure. On interactive tasks the target span is picked by a "stall signature", the most frequent recent command whose observation repeats or is empty.

The model then refills only the masked span, up to three times, and each fill is checked by a local probe (exact match, implicated unit tests, or on ALFWorld an admissible unused interact command). A skill is written only if the probe passes (or, on text and code, if there is a local fix), after names and IDs are scrubbed. At test time a router injects a family skeleton plus one or two matched skills. Experiments use a frozen Qwen3.5-4B on HotpotQA, GSM8K, HumanEval, MBPP, ALFWorld and AppWorld against ten baselines with ablation and cost tables, transfer to four other students, a teacher swap, and zero-shot transfer to four held-out QA sets. The main result is ALFWorld 44.0 vs 31.3 for the best baseline.

Critical strengths
  1. The span-level repair idea is simple and useful for long-horizon agents. Instead of regenerating the whole episode after a failure, only the stalled command is masked and refilled, and the stall signature (Eq. 4) is read from the environment observations, so no second judge model is needed. I think this matters because cost is the main problem of reflection-style methods on interactive tasks, and the cost tables support it: on AppWorld eval CAVRA uses 16.04 calls / 60.7k tokens vs 33.32 to 41.48 calls for the baselines in Table 8, and the evolve-side token bill is also the lowest on ALFWorld and AppWorld (Table 9).

  2. The ALFWorld gain is large and the protocol around it is mostly clean. Evolve and eval splits are disjoint (Table 1), the first-pass verdict is never rewritten, eval is a single student pass, and the ALFWorld margin (44.0±2.1 vs 31.3) is well outside CAVRA's own std. This is the main reason I think there is something real here, and why I am not at clear reject.

Critical weaknesses

The ALFWorld headline is not attributed to the claimed mechanism. Table 7 reports only 2 accesses on ALFWorld, so at most 2 skills were written, yet the score goes from 29.9 (w/o Evolve) to 44.0. At eval the router also injects a family skeleton, which line 316 says is distilled. In the released code these skeletons are eight hand-written recipes (cavra/skills.py, FAMILY_SKELETON, e.g. "heat it with a microwave, then put it in the asked receptacle"), and nothing distills them. The released ALFWorld evolve split has only pick and look tasks, while 92 of the 134 eval games are heat, cool, clean or pick_two, so on most of the eval set the only family-specific guidance is the hand-written recipe.

The released episode loop also replaces the model's command with an unused interact command and adds a hint when it detects a stall (cavra_bench/envs/alfworld.py), which I could not find in the paper. No ablation removes the skeleton alone, and without baseline code I cannot tell whether baselines get the skeleton or the stall override.

To be fair, w/o RM also gets the skeleton and scores 29.9, so the skeleton alone may not explain the gain, but the gain cannot be attributed without this ablation. Finally, "verified" is not measured: the ALFWorld probe accepts any admissible unused interact command, and in Fig. 5 the accepted fix for "place a newspaper on the couch" is "close drawer 18", with an official score of 0. This matters because the paper's contribution is the remask and gated distillation mechanism, and the biggest number could come from hand-designed components.

Questions for authors
  1. The released code hard-codes the family skeletons (cavra/skills.py, FAMILY_SKELETON). Do the Table 2 baselines receive them? Please report CAVRA without the skeleton (store kept), and the strongest baselines with the same skeleton added. If CAVRA without the skeleton still beats the best baseline on ALFWorld by clearly more than its std, and the baselines given the skeleton stay below CAVRA, I would raise my rating to 3.

  2. Does the stall override and hint in the released ALFWorld episode loop (cavra_bench/envs/alfworld.py) also apply to the baselines and to the w/o Evolve row?

  3. Are the Table 3 ALFWorld rows computed on all 134 eval games? cavra/config.py defines ABLATION_LIMITS with 67 ALFWorld games, all five ALFWorld ablation values are multiples of 1/67, and the first 67 eval games contain no heat, cool or pick_two task.

Reviewer_D6p5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes CAVRA, a textual self-distillation method for self-evolving agents with the frozen-model. The main claim is the current methods are at episode level repair and write the next policy from the raw transcript. CAVRA instead asks the same frozen model to localize the smallest span that cause the verifier to fail. Then the student fills only the masked span. A local probe checks if this makes progress. A skill is written only when the local check passes or a real local fix is found. The skill store is routed to later tasks by task family and lexical overlap. The experiments covers question answering, mathematics, coding, household control and app control. The paper reports improvement over ten baselines and a large gain on ALFWorld. It also reports ablations, cost, teacher swaps and cross-domain transfer.

Critical strengths
  1. is the local repair design. It addresses a real cost problem in self-evolving agents. Rewriting a whole episode after one failed step is wasteful and can inject new errors. CAVRA narrows the update to a small text span. This is conceptually cleaner than another full reflection pass.

  2. The experiment scope is broad. Six benchmarks and a separate cost analysis give readers a useful picture of where the method helps. The ALFWorld gain is relatively significant. The case study also makes the mechanism more concrete.

Critical weaknesses

The main concern is verifier telemetry leakage. The paper claims that hidden tests and gold labels never enter the teacher call or the skill distillation step. The method also uses implicated unit tests as local probes on code tasks. The paper does not state which tests are visible for each benchmark. If hidden tests reach the teacher or the probe, the gains may come from test access rather than from transferable skill induction. The claim is that it improves later tasks without editing the official score.

The second is comparison fairness. CAVRA spends extra calls on failures. The baselines mostly do not. Table 8 reports lower evaluation cost, but evaluation never enters the repair loop, so this does not answer the evolution cost question(correct me if I'm wrong). The paper needs a fixed budget comparison. Several reported gaps are also close to the run-to-run variance.

Questions for authors
  1. For each benchmark, what exact verifier information does the teacher see in Locate and what exact signal does the probe use? Does any hidden test, gold label, reference answer or judge output enter these calls?

  2. Under the same total call, token and verifier query budget, how does CAVRA compare with the strongest baseline? Please report a matched budget result.

  3. It shows gain on HumanEval with stronger teachers. Why does the paper conclude weak dependence on teacher scale? I want to see the teacher localization quality and student fill success.

  4. CAVRA adds a target-format check while other methods do not. Is this check also given to baselines? If not, can the cross-domain gain be attributed to the distilled skills?

  5. What is the measured instance leakage rate after scrubbing? Provide a manual audit of stored skills and evidence that contaminated skills do not harm later tasks.

  6. Table 3 shows that removing the loop lowers ALFWorld below the no-evolution baseline. What does it mean?

Reviewer_CmT4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses two chanlleges in the self-evolving of LLM agent: First, the agent regenerates the entire multi-turn trajectory if fails. Second, the agent writes prompt policies directly from the raw transcripts which haven't been checked. Both consume tokens and the same errors may occur again in later tasks.

The authors propose an on-policy textual self-distillation setup which converts failed traces into reusable skills without changing model weights. CAVRA uses the environment signature to figure out which command actually caused the failure, then masks just that one. The same frozen model then fills in the masked span, and a small local probe aligned to task progress checks whether the repair really changed the state in a meaningful way.

During evaluation, a matched Qwen 3.5 4B backbone is used on math, code synthesis, QA, and interactive control benchmarks. The proposed approach outperforms existing memory and reflection baseline, achieving a headline 40.6% gain in success rate on ALFWorld while cutting token useage.

Critical strengths

Rather than rerunning the whole multi-step trace whenever goes wrong, CAVRA investigates the environment’s own stall signature to figure out which command failed, and then fixes only the failure span.

The framework requires every local repair to pass the progress check first, and removes any task-specific names or IDs before they are saved.

The design makes the stored context as a rule and carries over to other tasks, rather than a replay of one particular episode. That’s a key factor to get a 40.6% relative gain on ALFWorld, allowing transfer to the new domain without training.

Critical weaknesses

In the paper, CAVRA does not require human judges, however, progress here is defined by rules tailored to each benchmark, which may lead to per-benchmark engineering. Moreover, it’s hard to see how this carries over to open ended settings like complicated tool use, where it not easy to get a clean signature or a local progress check.

All the reported gains come from single evolve pass. Figure 3 shows that with more rounds, on harder benchmarks like AppWorld performance falls off noticeably because the agent overfits to a narrow slice of tasks. I was wondering maybe the memory cap doesn’t help here.

Questions for authors

The method seems to rely on having clean test suites for code and a fixed command menu for environments like ALFWorld, which tell whether an action made progress. How does that transfer to open-ended, real-world tasks, where there’s no command list to check and no unit test that runs?

Figure 3 shows that running several evolution rounds on AppWorld actually makes worse, since the agent starts overfitting. With the memory store hard-capped at just 6 skills, how is the agent supposed to keep learning over time?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.