- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces TextualRL, a framework for optimizing the shared textual context of a frozen LLM agent using execution feedback. Its two principal components are: (1) Groupwise Policy Critique, which compares successful and failed executions of the same task and uses cross-task analysis when outcomes are uniform; and (2) Cross-Group Policy Update, which reviews proposed edits against independently collected trajectory evidence, including observations that produce no edits. The resulting bounded patch is accepted only when it improves the recorded held-out validation score. The paper evaluates five target models across six benchmarks and reports improvements over the tested baselines.
Critical strengths
- The paper identifies a concrete limitation of proposal-only aggregation: a trajectory can contain valuable evidence even when it does not justify an edit. This separation between proposing changes and preserving evidence is the paper’s most compelling contribution.
- The empirical evaluation is reasonably broad, and the ablations examine the proposed mechanisms.
Critical weaknesses
- The reinforcement-learning framing needs sharper qualification. The method performs reward-guided search over textual contexts, but it does not introduce a policy-gradient objective or a formal credit-assignment mechanism. The connection to GRPO is primarily an analogy based on grouped sampling. The paper does not sufficiently clarify what its RL formulation contributes beyond an existing reward-guided text optimization framework.
- Limited novelty. The space of train-free, text-space context optimization is already crowded (ACE, GEPA, SkillOpt, Trace2Skill, EvoSkill), and the training-loop skeleton of TextualRL corresponds item-by-item to SkillOpt, such as validation-gated acceptance and rejected-update memory. The main additions of TextualRL are the organization of rollout evidence and the explicit evidence-review stage. While SkillOpt substantially improves over the base model, TextualRL provides comparatively modest additional gains over SkillOpt in Table 1. These gains also come with substantially higher token usage on several benchmarks. For example, on ALFWorld with Qwen3.8-27B, TextualRL uses approximately three times as many recorded iteration tokens while improving limited accuracy. Without budget-matched comparisons, it remains unclear whether the proposed mechanisms offer a favorable performance–cost trade-off.
- The same validation set is repeatedly used to accept or reject context updates, and rejection feedback informs subsequent proposals. This creates a risk of adaptive overfitting to the validation set, particularly given the small validation splits used in several benchmarks.
- Some ablations confound the proposed mechanism with other changes. Removing outcome routing also changes analysis minibatches from task blocks to individual trajectories and modifies the analyst prompt. The resulting performance difference cannot be attributed exclusively to outcome routing.
- The experimental analysis provides limited insight into why the design decisions work. Much of the discussion restates numerical results from the tables rather than explaining the mechanisms underlying the improvements. Analyses do not sufficiently clarify when and why evidence review is beneficial or distinguish its contribution from the effects of additional sampling and optimizer computation. A clearer account of these mechanisms would help readers understand the rationale behind the method and the conditions under which it is effective.
Questions for authors
- What are the key algorithmic differences between SkillOpt and TextualRL, and which experimental results isolate the benefits of those differences?
- Were hyperparameters and design choices also selected using this validation set? If so, how was the resulting risk of overfitting controlled?
- Other weaknesses listed above.
4 comments
Response to Reviewer hihA (Part 2/2)
W4: Removing outcome routing also changes the analysis minibatches from task blocks to individual trajectories and modifies the analyst prompt, so the ablation cannot be attributed to routing alone.
A4: We agree that the original ablation bundles these changes. We added a task-blocked, unrouted variant. It forms minibatches of up to eight complete task blocks and keeps each trajectory associated with its originating task. Like the original unrouted ablation, it does not separate blocks by outcome category and uses the same unified analyst prompt. The two unrouted variants therefore differ only in the minibatch unit. The evidence-card instructions are retained.
| SearchQA | Spreadsheet | OfficeQA | DocVQA | LiveMath | ALFWorld | Avg. | |
|---|---|---|---|---|---|---|---|
| w/o outcome routing (trajectory minibatches) | 81.8 | 58.9 | 46.5 | 92.5 | 53.6 | 83.6 | 69.5 |
| Task-blocked, unrouted (new) | 81.5 | 59.3 | 47.1 | 92.2 | 54.4 | 83.6 | 69.7 |
| TextualRL (full) | 82.7 | 60.7 | 47.7 | 94.4 | 56.0 | 84.3 | 71.0 |
The task-blocked variant recovers 0.2 points, while the full method remains 1.3 points higher and improves all six benchmarks. Thus, block fragmentation alone does not account for the observed gap. The remaining difference from the full method includes outcome-aware grouping and its route-specific instructions, and the comparison does not separate these components. These are single-run ablations. We have added the new comparison to Table 2 and its precise grouping rule to Appendix B.2.
W5: The analysis provides limited insight into why the design works. It mostly restates numbers, does not clarify when and why evidence review helps, and does not separate its contribution from additional sampling and optimizer computation.
A5: We have organized the analysis around the information supplied by each route and how review uses it (Section 4.3 and Appendices B.6 and C.6).
-
Which outcomes the routes cover. Mixed groups account for 51% on LiveMath, giving many opportunities for within-task comparison. On SearchQA, DocVQA, and ALFWorld, 65–83% of groups are all-success, making cross-task preservation relevant to a large share of the collected data. OfficeQA has the highest all-failure share at 36%. These distributions describe the opportunities for each route, rather than measuring their causal contributions. We have added the per-epoch breakdown to Appendix C.6 (Table 15).
-
Which routes supply selected edits and cited evidence. The provenance analysis covers 78 selected edits in accepted candidates. We could match 29 selected edits to original analyst proposals. Of these, 21/29 (72%) came from the cross-task routes and 8/29 from the mixed route. The remaining 49 could not be assigned uniquely after consolidation, so the 72% applies only to the traceable subset. Review recorded 126 edit–card citation links for selected edits. Cross-task routes supplied 68/126 (54%), comprising 41 all-success and 27 all-failure links, while mixed groups supplied 58. These are citation links, not unique cards or counts of independently beneficial edits. They show that both the selected proposals and their review evidence draw on the cross-task routes.
-
How evidence can change applicability. Table 10 compares unreviewed and reviewed updates starting from the same historical context and evaluates them on the same test set. In the three examples, review adds 0.64–1.16 points over the unreviewed update. Figure 5 illustrates two mechanisms: a success from another group limits an over-general repair, and a failure restricts an overly permissive exception. These cases show how review can improve an update by adjusting its applicability. We have connected these examples to the quantitative comparisons in Appendix B.6, without treating them as an estimate of the frequency of each mechanism.
-
What the ablations isolate. The card ablations preserve the critique and review stages while restricting the evidence available to review, reducing accuracy by 1.2–1.7 points. SkillOpt (K=4) additionally tests increased sampling. Together, these comparisons support the importance of how trajectories are organized and used as evidence. G1–G2 distinguish these controls from a strict computation-matched experiment.
Dear Reviewer hihA,
Thank you again for your thoughtful review of TextualRL. We have posted our point-by-point response to your comments, including results from the additional experiments we ran in response to reviewer feedback. The changes are highlighted in the revised manuscript for ease of reference.
When you have a chance, we would appreciate hearing whether our response addresses your main concerns. If any questions remain, we would be happy to discuss them further.
Best regards, The authors of TextualRL
Response to the authors
Thank you for the detailed clarifications and additional experiments. The K=4 comparison and evidence-card ablations address some of my concerns. However, my main reservation remains the significance of the incremental contribution over SkillOpt, given that the outer optimization loop is inherited. Furthermore, although the provenance analysis and selected examples help explain how review operates, they provide limited evidence about when it reliably improves updates. Overall, the response narrows my concerns but does not sufficiently change my assessment of the contribution and its empirical support. I sincerely believe that stronger evidence on this point is important for establishing a substantial improvement over SkillOpt, and I encourage the authors to give this issue careful consideration. I therefore maintain my rating of 2 (Weak rejection).
Response to Reviewer hihA (Part 1/2)
Thank you for identifying the separation between proposing edits and preserving evidence as the central idea. We address the framing, experimental controls, and mechanism analysis below.
W1: The RL framing needs sharper qualification. The method has no policy-gradient objective or formal credit-assignment mechanism, and the connection to GRPO is mainly an analogy based on grouped sampling.
A1: We have distinguished the reward objective from the update procedure (Section 3.1). TextualRL collects trajectories under the current context and uses an optimizer LLM to propose textual edits, followed by validation-based selection. G3 formulates this as gradient-free, reward-driven policy optimization. The critique is a heuristic attribution procedure, without a policy-gradient estimator or convergence guarantee.
The correspondence to GRPO concerns the group-relative reward signal. Under the binary rewards used for routing, mixed groups have nonzero reward advantages, while uniform groups have zero reward advantages with the usual zero-variance handling. TextualRL can still extract preservation or repair evidence from uniform groups by comparing across tasks. This correspondence is restricted to GRPO's reward-advantage term. Uniform groups account for 74% on average across the six benchmarks, ranging from 49% to 90% (G3). We have made this scope explicit in Section 3.2.
W2 / Q1: The training-loop skeleton corresponds item by item to SkillOpt's. What are the key algorithmic differences, and which experiments isolate their benefits? Without budget-matched comparisons, is the performance–cost trade-off favorable?
A2: The outer loop is inherited from SkillOpt: validation-based acceptance, the edit-budget schedule, the rejected-edit buffer, and the slow/meta update. Our changes concern how each step constructs an update:
G1 summarizes the ablations. SkillOpt (K=4) scores 68.4, while TextualRL scores 71.0. Removing both cards and review gives 68.3. Restricting the cards available to review gives 69.3–69.8 while preserving the critique and review stages and their prompt templates. These comparisons support the roles of outcome-aware analysis and pooled evidence, beyond simply adding samples or a prompting stage.
With the same target, GPT-5.5 optimizer, and K=4, TextualRL achieves 70.97% at 318.18M iteration tokens versus 68.43% at 411.89M for SkillOpt (K=4), using 22.8% fewer recorded tokens. With K=2 (Table 4), TextualRL uses 1.37× SkillOpt's tokens and still scores 69.7 vs. 67.8.
W3 / Q2: The same validation set is used repeatedly to accept or reject updates, and rejection feedback informs later proposals, which risks adaptive overfitting on small validation splits. Were hyperparameters and design choices also selected on this validation set, as checkpoints are, and then evaluated on the test set? If so, how was the resulting risk controlled?
A3: The validation set is used only within each optimization run, to accept updates and select the checkpoint, which is then evaluated once on the test set. Hyperparameters and design choices were fixed before the main experiments and were not selected on either the validation or the test set:
Within a run, we agree that repeated selection on a small validation set can overfit that split, and the larger validation than test gains on OfficeQA and LiveMath (Section 4.4) reflect this. The split sizes follow SkillOpt's official release and are identical for all compared methods. Because the test split plays no role in any of these decisions, the reported test scores provide an independent assessment of the selected contexts. Test splits are about 7× larger than validation splits, and on ALFWorld the test environments come from the unseen split (
valid_unseen).The optimizer receives aggregate validation feedback through the within-epoch rejected-edit buffer, which resets each epoch. It does not receive validation inputs, trajectories, answers, or per-item outcomes. The feedback also informs the existing screen for edit families that occur in at least three rejected candidates and no accepted candidate.