acceptodds
Under review as a conference paper at ICLR 2027

TextualRL: Textual Reinforcement Learning for Context Management

Abstract

Textual context optimization allows LLM agents to adapt to task feedback without changing their parameters. However, outcome labels alone do not identify which behaviors to preserve or correct. Instructions inferred from individual trajectories can also overgeneralize across tasks, excluding behavior observed in successful executions under different conditions. To address these problems, we introduce TextualRL, a textual reinforcement learning framework that combines trajectory comparison with evidence-based refinement of context updates. Multiple Trajectory Analysis contrasts successful and failed trajectories within a task. Across tasks whose trajectories all succeed or all fail, it identifies behaviors worth preserving or formulates repair hypotheses, respectively. Parallel analysts produce proposed edits and evidence cards. Cross Group Refinement reviews these edits against relevant cards pooled across groups, retaining, narrowing, or revising instructions based on supporting observations and exceptions. Reviewed edits form a textual patch, and held-out validation determines whether to adopt the resulting context. We evaluate TextualRL with five target agents on six benchmarks. Averaged over the six benchmarks, TextualRL improves test accuracy over the no-skill baseline by 24.4 and 15.3 percentage points for GPT-5.5 and Qwen3.8-27B.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Agent, self-evolving, context management

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hihAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TextualRL, a framework for optimizing the shared textual context of a frozen LLM agent using execution feedback. Its two principal components are: (1) Groupwise Policy Critique, which compares successful and failed executions of the same task and uses cross-task analysis when outcomes are uniform; and (2) Cross-Group Policy Update, which reviews proposed edits against independently collected trajectory evidence, including observations that produce no edits. The resulting bounded patch is accepted only when it improves the recorded held-out validation score. The paper evaluates five target models across six benchmarks and reports improvements over the tested baselines.

Critical strengths
  1. The paper identifies a concrete limitation of proposal-only aggregation: a trajectory can contain valuable evidence even when it does not justify an edit. This separation between proposing changes and preserving evidence is the paper’s most compelling contribution.
  2. The empirical evaluation is reasonably broad, and the ablations examine the proposed mechanisms.
Critical weaknesses
  1. The reinforcement-learning framing needs sharper qualification. The method performs reward-guided search over textual contexts, but it does not introduce a policy-gradient objective or a formal credit-assignment mechanism. The connection to GRPO is primarily an analogy based on grouped sampling. The paper does not sufficiently clarify what its RL formulation contributes beyond an existing reward-guided text optimization framework.
  2. Limited novelty. The space of train-free, text-space context optimization is already crowded (ACE, GEPA, SkillOpt, Trace2Skill, EvoSkill), and the training-loop skeleton of TextualRL corresponds item-by-item to SkillOpt, such as validation-gated acceptance and rejected-update memory. The main additions of TextualRL are the organization of rollout evidence and the explicit evidence-review stage. While SkillOpt substantially improves over the base model, TextualRL provides comparatively modest additional gains over SkillOpt in Table 1. These gains also come with substantially higher token usage on several benchmarks. For example, on ALFWorld with Qwen3.8-27B, TextualRL uses approximately three times as many recorded iteration tokens while improving limited accuracy. Without budget-matched comparisons, it remains unclear whether the proposed mechanisms offer a favorable performance–cost trade-off.
  3. The same validation set is repeatedly used to accept or reject context updates, and rejection feedback informs subsequent proposals. This creates a risk of adaptive overfitting to the validation set, particularly given the small validation splits used in several benchmarks.
  4. Some ablations confound the proposed mechanism with other changes. Removing outcome routing also changes analysis minibatches from task blocks to individual trajectories and modifies the analyst prompt. The resulting performance difference cannot be attributed exclusively to outcome routing.
  5. The experimental analysis provides limited insight into why the design decisions work. Much of the discussion restates numerical results from the tables rather than explaining the mechanisms underlying the improvements. Analyses do not sufficiently clarify when and why evidence review is beneficial or distinguish its contribution from the effects of additional sampling and optimizer computation. A clearer account of these mechanisms would help readers understand the rationale behind the method and the conditions under which it is effective.
Questions for authors
  1. What are the key algorithmic differences between SkillOpt and TextualRL, and which experimental results isolate the benefits of those differences?
  2. Were hyperparameters and design choices also selected using this validation set? If so, how was the resulting risk of overfitting controlled?
  3. Other weaknesses listed above.
4 comments
AuthorsOpenReview ↗

Response to Reviewer hihA (Part 1/2)

Thank you for identifying the separation between proposing edits and preserving evidence as the central idea. We address the framing, experimental controls, and mechanism analysis below.

W1: The RL framing needs sharper qualification. The method has no policy-gradient objective or formal credit-assignment mechanism, and the connection to GRPO is mainly an analogy based on grouped sampling.

A1: We have distinguished the reward objective from the update procedure (Section 3.1). TextualRL collects trajectories under the current context and uses an optimizer LLM to propose textual edits, followed by validation-based selection. G3 formulates this as gradient-free, reward-driven policy optimization. The critique is a heuristic attribution procedure, without a policy-gradient estimator or convergence guarantee.

The correspondence to GRPO concerns the group-relative reward signal. Under the binary rewards used for routing, mixed groups have nonzero reward advantages, while uniform groups have zero reward advantages with the usual zero-variance handling. TextualRL can still extract preservation or repair evidence from uniform groups by comparing across tasks. This correspondence is restricted to GRPO's reward-advantage term. Uniform groups account for 74% on average across the six benchmarks, ranging from 49% to 90% (G3). We have made this scope explicit in Section 3.2.

W2 / Q1: The training-loop skeleton corresponds item by item to SkillOpt's. What are the key algorithmic differences, and which experiments isolate their benefits? Without budget-matched comparisons, is the performance–cost trade-off favorable?

A2: The outer loop is inherited from SkillOpt: validation-based acceptance, the edit-budget schedule, the rejected-edit buffer, and the slow/meta update. Our changes concern how each step constructs an update:

  1. Multiple rollouts per task under the same context, organized by their observed outcome pattern.
  2. Evidence cards produced independently of whether an analyst proposes an edit.
  3. Evidence-conditioned final consolidation, replacing the failure-first proposal merge.

G1 summarizes the ablations. SkillOpt (K=4) scores 68.4, while TextualRL scores 71.0. Removing both cards and review gives 68.3. Restricting the cards available to review gives 69.3–69.8 while preserving the critique and review stages and their prompt templates. These comparisons support the roles of outcome-aware analysis and pooled evidence, beyond simply adding samples or a prompting stage.

With the same target, GPT-5.5 optimizer, and K=4, TextualRL achieves 70.97% at 318.18M iteration tokens versus 68.43% at 411.89M for SkillOpt (K=4), using 22.8% fewer recorded tokens. With K=2 (Table 4), TextualRL uses 1.37× SkillOpt's tokens and still scores 69.7 vs. 67.8.

W3 / Q2: The same validation set is used repeatedly to accept or reject updates, and rejection feedback informs later proposals, which risks adaptive overfitting on small validation splits. Were hyperparameters and design choices also selected on this validation set, as checkpoints are, and then evaluated on the test set? If so, how was the resulting risk controlled?

A3: The validation set is used only within each optimization run, to accept updates and select the checkpoint, which is then evaluated once on the test set. Hyperparameters and design choices were fixed before the main experiments and were not selected on either the validation or the test set:

  • Batch size, analysis and merge minibatch sizes, the edit-budget schedule, epochs, and rollout temperature follow SkillOpt's defaults.
  • The 30 target–benchmark pairs share the same method hyperparameters. Environment-specific execution settings are documented separately.
  • K=4 was fixed before the K=1/2/4/8 analysis. We have corrected the sentence “so we use it as the default setting,” which suggested the reverse order (Section 4.4).
  • Prompts were developed using training-split trajectories only.

Within a run, we agree that repeated selection on a small validation set can overfit that split, and the larger validation than test gains on OfficeQA and LiveMath (Section 4.4) reflect this. The split sizes follow SkillOpt's official release and are identical for all compared methods. Because the test split plays no role in any of these decisions, the reported test scores provide an independent assessment of the selected contexts. Test splits are about 7× larger than validation splits, and on ALFWorld the test environments come from the unseen split (valid_unseen).

The optimizer receives aggregate validation feedback through the within-epoch rejected-edit buffer, which resets each epoch. It does not receive validation inputs, trajectories, answers, or per-item outcomes. The feedback also informs the existing screen for edit families that occur in at least three rejected candidates and no accepted candidate.

AuthorsOpenReview ↗

Response to Reviewer hihA (Part 2/2)

W4: Removing outcome routing also changes the analysis minibatches from task blocks to individual trajectories and modifies the analyst prompt, so the ablation cannot be attributed to routing alone.

A4: We agree that the original ablation bundles these changes. We added a task-blocked, unrouted variant. It forms minibatches of up to eight complete task blocks and keeps each trajectory associated with its originating task. Like the original unrouted ablation, it does not separate blocks by outcome category and uses the same unified analyst prompt. The two unrouted variants therefore differ only in the minibatch unit. The evidence-card instructions are retained.

SearchQASpreadsheetOfficeQADocVQALiveMathALFWorldAvg.
w/o outcome routing (trajectory minibatches)81.858.946.592.553.683.669.5
Task-blocked, unrouted (new)81.559.347.192.254.483.669.7
TextualRL (full)82.760.747.794.456.084.371.0

The task-blocked variant recovers 0.2 points, while the full method remains 1.3 points higher and improves all six benchmarks. Thus, block fragmentation alone does not account for the observed gap. The remaining difference from the full method includes outcome-aware grouping and its route-specific instructions, and the comparison does not separate these components. These are single-run ablations. We have added the new comparison to Table 2 and its precise grouping rule to Appendix B.2.

W5: The analysis provides limited insight into why the design works. It mostly restates numbers, does not clarify when and why evidence review helps, and does not separate its contribution from additional sampling and optimizer computation.

A5: We have organized the analysis around the information supplied by each route and how review uses it (Section 4.3 and Appendices B.6 and C.6).

  1. Which outcomes the routes cover. Mixed groups account for 51% on LiveMath, giving many opportunities for within-task comparison. On SearchQA, DocVQA, and ALFWorld, 65–83% of groups are all-success, making cross-task preservation relevant to a large share of the collected data. OfficeQA has the highest all-failure share at 36%. These distributions describe the opportunities for each route, rather than measuring their causal contributions. We have added the per-epoch breakdown to Appendix C.6 (Table 15).

  2. Which routes supply selected edits and cited evidence. The provenance analysis covers 78 selected edits in accepted candidates. We could match 29 selected edits to original analyst proposals. Of these, 21/29 (72%) came from the cross-task routes and 8/29 from the mixed route. The remaining 49 could not be assigned uniquely after consolidation, so the 72% applies only to the traceable subset. Review recorded 126 edit–card citation links for selected edits. Cross-task routes supplied 68/126 (54%), comprising 41 all-success and 27 all-failure links, while mixed groups supplied 58. These are citation links, not unique cards or counts of independently beneficial edits. They show that both the selected proposals and their review evidence draw on the cross-task routes.

  3. How evidence can change applicability. Table 10 compares unreviewed and reviewed updates starting from the same historical context and evaluates them on the same test set. In the three examples, review adds 0.64–1.16 points over the unreviewed update. Figure 5 illustrates two mechanisms: a success from another group limits an over-general repair, and a failure restricts an overly permissive exception. These cases show how review can improve an update by adjusting its applicability. We have connected these examples to the quantitative comparisons in Appendix B.6, without treating them as an estimate of the frequency of each mechanism.

  4. What the ablations isolate. The card ablations preserve the critique and review stages while restricting the evidence available to review, reducing accuracy by 1.2–1.7 points. SkillOpt (K=4) additionally tests increased sampling. Together, these comparisons support the importance of how trajectories are organized and used as evidence. G1–G2 distinguish these controls from a strict computation-matched experiment.

Dear Reviewer hihA,

Thank you again for your thoughtful review of TextualRL. We have posted our point-by-point response to your comments, including results from the additional experiments we ran in response to reviewer feedback. The changes are highlighted in the revised manuscript for ease of reference.

When you have a chance, we would appreciate hearing whether our response addresses your main concerns. If any questions remain, we would be happy to discuss them further.

Best regards, The authors of TextualRL

Reviewer_hihAOpenReview ↗

Response to the authors

Thank you for the detailed clarifications and additional experiments. The K=4 comparison and evidence-card ablations address some of my concerns. However, my main reservation remains the significance of the incremental contribution over SkillOpt, given that the outer optimization loop is inherited. Furthermore, although the provenance analysis and selected examples help explain how review operates, they provide limited evidence about when it reliably improves updates. Overall, the response narrows my concerns but does not sufficiently change my assessment of the contribution and its empirical support. I sincerely believe that stronger evidence on this point is important for establishing a substantial improvement over SkillOpt, and I encourage the authors to give this issue careful consideration. I therefore maintain my rating of 2 (Weak rejection).

Reviewer_ULn4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TextualRL, a method that performs context management for large language model (LLM) agents without model parameter update. More specifically, TextualRL uses Groupwide Policy Critique that contrasts successful and failed rollouts within a task. Then, TextualRL performs Cross-Group Policy Update consisting of evidence-based proposal review and validation-gated updates.

Critical strengths
  • S1. The proposed method is simple and practical.

  • S2. This paper provides comprehensive experiment results using five target agents (e.g., GPT-5.5, GPT-5.4, Qwen3.5-4B, Qwen3.6-35B-A3B, and Qwen3.8-27B) on six benchmarks (e.g. SearchQA, Spreadsheet, OfficeQA, DocVQA, LiveMath, and ALFWorld).

Critical weaknesses
  • W1. Performance gain of TextualRL seems somewhat modest compared to recent works such as SkillOpt. More specifically, according to Table 1, TextualRL can compare with SkillOpt as follows: | Model | Skill Source | SearchQA | LiveMath | ALFWorld | | - | - | - | - | - | | GPT-5.5 | SkillOpt | 87.3 | 66.9 | 95.5 | | GPT-5.5 | TextualRL | 88.9 | 67.2 | 95.5 | | Qwen3.5-4B | SkillOpt | 71.2 | 52.0 | 81.3 | | Qwen3.5-4B | TextualRL | 73.9 | 56.0 | 82.1 |

  • W2. Although this paper comprehensively compares TextualRL with recent works, the key contribution of TextualRL does not seem to be clearly stated compared to those works.

  • W3. I am not sure that the proposed method can be considered as reinforcement learning. If it is, the problem formulation needs to be clearly provided.

  • W4. The basic assumption of Groupwise Policy Critique is that there are reusable and shared contexts across tasks. However, this kind of assumption on task structure may limit the application of TextualRL.

Questions for authors
  • Q1. What is the main difference between TextualRL and SkillOpt?
4 comments
AuthorsOpenReview ↗

Response to Reviewer ULn4 (Part 1/2)

Thank you for highlighting the practicality of the method and the breadth of the evaluation. We address the size of the gain, the technical contribution, the RL framing, and the shared-context assumption below.

W1: The performance gain of TextualRL seems modest compared with recent work such as SkillOpt.

A1: The gains are modest in several settings, but their direction is consistent across target models. TextualRL improves over SkillOpt in 29 of the 30 target–benchmark settings and ties in the remaining one. On Qwen3.8-27B, the six-benchmark averages across three independent optimization runs per method are 70.68% for TextualRL and 67.43% for SkillOpt, a difference of 3.25 percentage points. The mean improvement is positive on all six benchmarks.

Limited headroom explains some small gains, such as GPT-5.5 on DocVQA and ALFWorld, where SkillOpt already scores 91.2% and 95.5%. It does not explain every setting. We therefore report both the absolute improvements and their cost. G2 presents the comparison with SkillOpt using the same K = 4 rollouts per task and distinguishes sampling count from measured token consumption.

The comparison in W1 appears to have been cut off after “as follows:”. Please let us know if you had particular settings in mind.

W2 / Q1: What is the main difference between TextualRL and SkillOpt? The key contribution relative to recent work is not clearly stated.

A2: TextualRL changes how rollout evidence is organized and used to construct an update. It retains SkillOpt's outer loop, including validation-gated acceptance, the edit-budget schedule, the rejected-edit buffer, and the slow/meta update. Our changes are:

  1. Outcome-conditioned analysis. We collect K rollouts of each task under the same context. Mixed outcomes support within-task contrast, while all-success and all-failure tasks are analyzed across tasks for preservation and repair.
  2. Evidence decoupled from proposals. Critics can return evidence cards even when their trajectories do not justify an edit. These observations remain available when other groups propose changes.
  3. Evidence-conditioned consolidation. The final merge reviews candidate rules against the pooled cards to determine their content and applicability, replacing SkillOpt's fixed failure-first priority.

G1 maps these changes to the ablations. In particular, restricting review to originating-group cards or removing cards from groups without edits reduces the average by 1.2 and 1.7 points. These variants preserve the prompting stages while changing evidence access. They support the contribution of pooled evidence beyond simply adding another prompting stage. We have used this distinction to sharpen the comparison with prior work in Section 2.

W3: Can the proposed method be considered reinforcement learning? If so, the problem formulation needs to be stated clearly.

A3: We have qualified the framing as gradient-free, reward-driven policy optimization in context space. The target model's weights remain fixed. Its context-conditioned policy generates on-policy training trajectories, and the optimizer proposes textual changes to the context to improve expected task reward. A candidate is accepted only when its recorded held-out validation score improves. This is an empirical selection rule, not a guarantee of improvement in expected reward.

The GRPO connection concerns the within-group reward signal. With the binary success rewards used here, mixed groups have nonzero normalized reward advantages, while all-success and all-failure groups have zero reward advantages. TextualRL uses cross-task analysis for the latter groups. This correspondence does not imply that every term in a GRPO update, such as regularization, vanishes, nor does it establish equivalence to policy-gradient optimization. G3 gives the formulation and its limits. We have made these qualifications explicit in the abstract, the introduction, and Sections 3.1–3.2.

AuthorsOpenReview ↗

Response to Reviewer ULn4 (Part 2/2)

W4: Groupwise Policy Critique assumes a reusable context shared across tasks, which may limit the applicability of TextualRL.

A4: TextualRL assumes that tasks share useful regularities, but they need not require identical behavior. A context is optimized separately for each target–benchmark pair. Conditional instructions can express both common procedures and task-specific behavior within that context. The cross-task routes seek support from at least two distinct tasks, and evidence review can narrow a proposed rule to the conditions supported by the trajectories.

To examine this assumption, we broke down the Qwen3.8-27B ALFWorld results by task type (Appendix C.6). All 134 test environments use one learned context. The initial-context column averages three saved evaluations of the same initial context, while the TextualRL column is one evaluation of the selected context. The table lists the four task types whose accuracy changed.

Task type (n)Initial contextTextualRLΔ
Heat & place (23)59.478.3+18.8
Pick two & place (17)68.676.5+7.8
Examine in light (18)77.883.3+5.6
Pick & place (24)86.187.5+1.4

Scores are percentages and differences are percentage points. Differences are computed before rounding, so displayed values can differ by 0.1 when subtracted.

The remaining two task types are unchanged, so no task type loses accuracy. The largest observed improvements occur on heat-and-place and two-object tasks. The learned context combines shared procedures, such as completing a location sweep and avoiding repeated navigation loops, with conditional procedures for appliances and lamps. This illustrates how one context can accommodate heterogeneous tasks. We treat the per-type differences as descriptive observations, rather than a guarantee that a shared context cannot harm any task type. Stronger task heterogeneity may require multiple contexts or a routed skill library, which we now discuss in the limitations (Appendix D).

Dear Reviewer ULn4,

Thank you again for your thoughtful review of TextualRL. We have posted our point-by-point response to your comments, including results from the additional experiments we ran in response to reviewer feedback. The changes are highlighted in the revised manuscript for ease of reference.

When you have a chance, we would appreciate hearing whether our response addresses your main concerns. If any questions remain, we would be happy to discuss them further.

Best regards, The authors of TextualRL

Reviewer_ULn4OpenReview ↗

Reviewer Response to the Author Responses

Thank you for providing thoughtful responses to my comments and questions. Especially, I appreciate the comparison between TextualRL and SkillOpt. Although TextualRL has some changes, its contributions seem somewhat limited compared to SkillOpt. Therefore, for now, I maintain my initial rating of 2.

Reviewer_GPoQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how LLM agents can learn reusable instructions from task feedback without updating their model weights. The paper proposes TextualRL, which generates several attempts at each training task and routes each task by its outcome pattern. Successful and failed attempts are compared within mixed-outcome tasks, while all-success and all-failure tasks are analyzed across tasks to identify behaviors to preserve or failures to repair. A second stage reviews the proposed edits against evidence pooled from other groups, including observations that did not produce an edit. The reviewed edits form a bounded patch, which is kept only if it improves validation performance. Experiments evaluate the method on six benchmarks with five target models.

Critical strengths
  1. The paper addresses a practically meaningful problem. Learning reusable instructions from execution feedback offers a way to adapt LLM agents when updating their model weights is unavailable or impractical. Improving how this feedback is converted into instructions could reduce manual prompt engineering and help agents avoid repeating mistakes across tasks.

  2. The evaluation is broad and the direction of improvement is consistent. The experiments cover six benchmarks and five target models, and TextualRL matches or exceeds SkillOpt in all 30 model-benchmark combinations. In three independent optimization runs on Qwen3.8-27B, TextualRL stays ahead of SkillOpt in every run (3.18-3.36 points overall).

Critical weaknesses
  1. The method largely recombines components of existing methods. TextualRL compares successful and failed attempts at the same task, and extracts patterns shared by successful tasks. ExpeL already extracts insights from exactly these two sources. SkillOpt already learns from both successful and failed trajectories, merges the resulting edit proposals, and accepts bounded updates only when they improve validation performance. Relative to these two methods, TextualRL mainly adds a route for all-failure tasks and an LLM step that reviews merged proposals against evidence cards. The paper does not show that these additions amount to more than extra prompting stages.

  2. The gain over SkillOpt is small relative to the extra training cost. On Qwen3.8-27B, TextualRL consumes 2.05-3.24 times SkillOpt's tokens during context optimization on five of the six benchmarks (Table 8). In return, its six-benchmark average is 3.2 points higher (71.0% vs. 67.8%). On OfficeQA, for example, TextualRL uses about three times SkillOpt's tokens (150M vs. 49M) for a gain of 1.2 points (47.7% vs. 46.5%). Across all five target models, the average gain over SkillOpt is 0.9-3.2 points. With this cost-benefit ratio, it is unclear whether TextualRL is preferable to SkillOpt in practice.

  3. The ablation result for evidence review is inconsistent with the paper's own audit. The paper claims that evidence review improves the context by narrowing or revising over-general edits. Removing this review causes the largest drop among all ablations (2.7 points, Table 2). However, the audit in Table 9 shows that review narrows or revises only 37 of 436 edits (8.48%), and the other edits pass through unchanged. It is unclear how a step that leaves over 90% of edits unchanged can account for the largest drop. The paper therefore does not establish that this gain comes from the mechanism it describes.

Questions for authors
  1. ExpeL already learns from same-task success-failure pairs and from patterns shared by successful tasks, and SkillOpt already merges proposals from successful and failed trajectories into bounded, validation-gated updates. What does TextualRL contribute technically beyond these two methods?

  2. Appendix C.5.1 reuses the harness baselines reported by SkillOpt, and the ALFWorld validation set here (18 environments) differs from the 140 described by SkillOpt. Were the baselines in Table 1 rerun with the same optimizer, initial context, and data splits as TextualRL?

  3. TextualRL consumes 2.05-3.24 times SkillOpt's optimization tokens on five of the six benchmarks (Table 8). If SkillOpt is given the same token budget, does TextualRL still outperform it?

  4. Table 9 shows that evidence review narrows or revises only 8.48% of edits, yet removing it causes the largest ablation drop (2.7 points, Table 2). What explains this gap?

3 comments
AuthorsOpenReview ↗

Response to Reviewer GPoQ (Part 1/2)

Thank you for the detailed review. Your questions identify where the paper needs a sharper account of its contribution, comparison protocol, and evidence-review ablation.

W1 / Q1: ExpeL already learns from same-task success–failure pairs and patterns shared by successful tasks, and SkillOpt already merges successful and failed trajectories into bounded, validation-based updates. What does TextualRL add beyond these methods?

A1: The contribution is the organization of rollout evidence and its use during consolidation. G1 gives the comparison and ablation results.

  • ExpeL compares attempts generated with different accumulated reflections. TextualRL samples multiple attempts under the same current context, preserving a within-task comparison without intervening reflection updates. Tasks with no success among the current samples enter cross-task repair analysis.
  • SkillOpt's merge receives proposals. TextualRL also preserves evidence from analysts that propose no edit, allowing a successful behavior elsewhere to constrain a proposed repair.
  • The card ablations preserve the critique/review stages and prompt templates while restricting evidence access. Their 1.2- and 1.7-point drops support the value of the evidence available to review, beyond simply adding a prompting stage.

The new SkillOpt (K=4) run increases sampling to four rollouts per task and scores 68.43%, compared with TextualRL's 71.0% in Table 1. The task-blocked unrouted variant scores 69.7%. Together, these comparisons support outcome-conditioned analysis and pooled evidence as contributors to the full method's result. We have made these distinctions explicit in Section 2 and report both comparisons in Tables 2 and 4.

W2 / Q3: TextualRL uses substantially more optimization tokens for modest gains. If SkillOpt receives the same token budget, does TextualRL still outperform it?

A2: Yes. Even with 29% more iteration tokens than TextualRL, SkillOpt with the same four rollouts per task scores lower on all six benchmarks. SkillOpt (K=4) records 411.89M iteration tokens across six benchmarks, compared with TextualRL's 318.18M. It reaches 68.43% average accuracy, compared with TextualRL's 70.97%. Both runs use the same target and the GPT-5.5 optimizer. In the other direction, TextualRL with K=2 (Table 4) records 218.51M tokens, 1.37× SkillOpt's, and still scores 1.9 points higher (69.7 vs. 67.8). G2 reports the per-benchmark scores and token accounting.

The three independent runs per method provide complementary evidence about repeatability. Their averages are 70.68% for TextualRL and 67.43% for SkillOpt. Even selecting each benchmark's highest test score from the three completed SkillOpt runs yields 68.04%, below TextualRL's mean.

AuthorsOpenReview ↗

Response to Reviewer GPoQ (Part 2/2)

W3 / Q4: Evidence review narrows or revises only 8.48% of edits, yet removing it causes a 2.7-point drop. What explains the discrepancy?

A3: The ablation and audit measure different quantities. The ablation removes cards and evidence-conditioned consolidation together, whereas 8.48% counts explicit narrow/revise labels among edits returned by review. We have corrected the wording in Section 4.3 and renamed the ablation "w/o evidence cards & review" in Table 2.

  1. The ablation removes the evidence pathway. Critics no longer produce cards, and final consolidation returns to SkillOpt's failure-first proposal merge. The 2.7-point difference therefore measures this joint change, rather than the contribution of the 37 explicitly narrowed or revised edits alone.
  2. Review is the final evidence-conditioned merge. The optimizer receives the failure-side and success-side candidates together with pooled cards, then outputs the final edit set. Decisions about selection, combination, and applicability occur in this call. The narrow/revise rate does not enumerate every candidate-level effect of evidence.
  3. The update comparisons provide direct examples. Table 10 starts from the same historical context and compares an unreviewed consolidated proposal with the reviewed update on the same test set. The reviewed updates improve accuracy by 0.64–1.16 points in the three selected cases. Figure 5 illustrates how evidence narrows an over-general rule or restricts an overly permissive exception. These examples establish that such corrections can matter, without estimating how much of the full 2.7-point ablation difference they explain.

Changes can also affect later rollouts and the path of accepted contexts. Appendix B.6 now presents this as a possible compounding mechanism, rather than a measured attribution of the final gain. The evidence-access ablations in G1 provide the complementary comparison while retaining the review stage.

Q2: Were the baselines rerun with the same optimizer, initial context, and data splits? The paper reuses harness baselines, and its ALFWorld validation set has 18 environments rather than the 140 stated by SkillOpt.

A4: The provenance differs by target, and we have stated it explicitly in the implementation details (Section 4.1) and in the captions of Figure 3 and Table 11.

  • Four targets and harness baselines. For GPT-5.5, GPT-5.4, Qwen3.5-4B, and Qwen3.6-35B-A3B, the baseline rows in Table 1 are quoted from SkillOpt. The harness baselines are also quoted. These are comparisons with published results, not our own reruns.
  • Qwen3.8-27B. We independently ran the baseline experiments for this target, including the three SkillOpt runs. The main comparisons use the same splits, benchmark initial contexts, and evaluation protocol. The optimization-based baselines use GPT-5.5 as optimizer.
  • ALFWorld. We use the official released manifest: 39 training, 18 validation, and 134 test environments. The train, validation, and test game lists match that release item for item, and the validation manifest has contained 18 environments since its initial release. The paper text states 140, which differs from the manifest. The release establishes our reproduction protocol, but by itself does not establish which validation set was used to obtain the quoted paper results. Our Qwen3.8 comparisons use the same 18 validation environments for all methods.

Dear Reviewer GPoQ,

Thank you again for your thoughtful review of TextualRL. We have posted our point-by-point response to your comments, including results from the additional experiments we ran in response to reviewer feedback. The changes are highlighted in the revised manuscript for ease of reference.

When you have a chance, we would appreciate hearing whether our response addresses your main concerns. If any questions remain, we would be happy to discuss them further.

Best regards, The authors of TextualRL

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response to All Reviewers (Part 1/3)

We thank all three reviewers for their careful reading and constructive feedback. We appreciate their recognition of the practical problem, the breadth of the evaluation, and the separation between proposing edits and preserving evidence. We address the shared questions about novelty, cost, and the RL framing here, followed by replies to each reviewer. We have uploaded a revised manuscript in which all changes are highlighted in blue.

G1. What TextualRL adds beyond SkillOpt and ExpeL (GPoQ-Q1, ULn4-W2/Q1, hihA-W2/Q1)

Shared outer loop and technical contribution. TextualRL retains SkillOpt's validation-based acceptance, edit-budget schedule, within-epoch rejected-edit buffer, and epoch-level slow/meta update. Our changes concern how the current context generates rollout evidence and how that evidence is converted into an update.

ExpeLSkillOptTextualRL
Rollouts per taskReflexion retries conditioned on earlier reflectionsOne rolloutK independent samples under the same current context
Success–failure comparisonSuccessful and failed attempts with different accumulated reflectionsSuccesses and failures pooled separately across tasksWithin-task contrast under the same context
No success observed in the sampled attemptsNo successful counterpart for pairwise comparisonIncluded in failure minibatchesSeparate cross-task repair analysis, requiring support from at least two distinct tasks
Information available for consolidationInsights with importance scoresEdit proposalsProposals and pooled evidence cards, including observations that yield no edit
ConsolidationADD / EDIT / UPVOTE / DOWNVOTEFailure-first proposal mergeEvidence-conditioned review of edit content and applicability
Collection and updatingExperience gathering followed by insight extractionIterative collection and validation-based updatesIterative collection and validation-based updates

The grouping gives the critic information that an isolated failure does not provide. If three of four rollouts succeed, the current context can already elicit a successful behavior, and comparing the unsuccessful attempt with those successes can inform a corrective edit. If all four fail, there is no observed successful counterpart. The critic instead looks for a recurring failure mechanism across tasks. Four failures do not establish that the task is unsolvable, so the output is a repair hypothesis rather than a diagnosis guaranteed by the outcome labels.

The evidence pool addresses a different limitation of proposal-only merging. A successful trajectory may warrant no edit but still limit a rule proposed elsewhere. Figure 5a illustrates this with a successful answer that preserves the title "Dalai Lama", providing an exception to an over-general shortening rule.

Ablations and additional comparisons. The following averages use Qwen3.8-27B as target and cover six benchmarks. The new comparisons are SkillOpt (K=4) and the task-blocked unrouted variant.

ConfigurationSettingAvg.
SkillOptOriginal method, K=167.8
TextualRL w/o evidence cards & reviewK=4 and outcome routing, proposal merge68.3
TextualRL, K=1Cross-task routes, cards and review, no within-task contrast68.4
SkillOpt (K=4)Pooled success/failure trajectories, no cards or review68.4
TextualRL, remove cards without editsReview excludes evidence from groups that proposed no edit69.3
TextualRL w/o outcome routingIndividual trajectories in unified analyst minibatches69.5
TextualRL, task-blocked without routingComplete task blocks, no outcome routing, unified analysis69.7
TextualRL, intra-group cards onlyReview restricted to originating-group evidence69.8
TextualRL (full)Outcome routing and pooled evidence review71.0

These results support complementary roles for outcome-conditioned analysis and evidence access. Increasing SkillOpt's rollout count to four does not recover the full method's performance in this run. Preserving task blocks in the unrouted variant also leaves a 1.3-point gap. The two card ablations preserve the critique/review stages and prompt templates while restricting eligible evidence, yet lose 1.2 and 1.7 points. They therefore test the value of evidence access beyond the presence of an additional prompting stage.

AuthorsOpenReview ↗

General Response to All Reviewers (Part 2/3)

G2. Performance and recorded token usage (GPoQ-W2/Q3, ULn4-W1, hihA-W2)

Size and repeatability of the gain. Across three independent optimization runs per method, TextualRL averages 70.68% and SkillOpt 67.43%, with a positive mean difference on every benchmark. This is about a 10% reduction in SkillOpt's remaining error. Gains are smaller in some settings, including high-scoring GPT-5.5 results on DocVQA and ALFWorld, so we report the associated token usage alongside accuracy.

What the recorded cost includes. Table 8 reports 318.18M iteration tokens for TextualRL and 159.41M for SkillOpt, summed over six benchmarks, with Qwen3.8-27B as target and GPT-5.5 as optimizer for both methods. These are recorded step-level tokens, including candidate validation, rather than a complete accounting of every training-related call. Target-side calls dominate the increase. On OfficeQA, 142.87M of TextualRL's 150.08M tokens are target-side usage, while optimizer usage is 7.21M versus SkillOpt's 4.70M. On Spreadsheet, TextualRL's recorded total is lower, at 41.34M versus 54.60M. Target usage includes both training rollouts and validation, so its increase cannot be attributed entirely to K.

A control at the same rollout count. We trained SkillOpt with K=4 rollouts per task, the same as TextualRL. All sampled trajectories enter SkillOpt's success/failure minibatches. The run keeps SkillOpt's four epochs and outer-loop settings and uses GPT-5.5 as optimizer. This tests increased sampling within SkillOpt without adding outcome routing or evidence cards.

SearchQASpreadsheetOfficeQADocVQALiveMathALFWorldAvg.
SkillOpt K=1 (Table 1)80.456.446.593.348.082.167.8
SkillOpt K=4 (new)80.657.146.593.949.682.868.4
TextualRL K=4 (Table 1)82.760.747.794.456.084.371.0

The new SkillOpt (K=4) run achieves 68.43% and records 411.89M iteration tokens, comprising 346.19M target tokens and 65.70M optimizer tokens. More rollouts improve SkillOpt by 0.6 points, but with the same target, GPT-5.5 optimizer, and K=4, TextualRL achieves 70.97% with 318.18M tokens: 2.54 percentage points higher average accuracy with 22.8% fewer recorded iteration tokens across the six benchmarks. TextualRL remains higher on all six benchmarks. This single-run comparison supports a better observed accuracy–token trade-off and suggests that increasing rollout count alone does not account for the advantage. It controls rollout count, without enforcing identical per-benchmark token budgets.

Accuracy versus recorded tokens. The K=2 ablation in Table 4 records 218.51M iteration tokens under the same accounting. The four configurations span the observed trade-off.

ConfigurationTokens (M)Relative to SkillOptAvg. accuracy
SkillOpt (K=1)159.411.00×67.8
TextualRL (K=2)218.511.37×69.7
TextualRL (K=4, default)318.182.00×71.0
SkillOpt (K=4)411.892.58×68.4

With K=2, TextualRL uses 1.37× SkillOpt's tokens and 31% fewer than the default, and still exceeds SkillOpt by 1.9 points. SkillOpt (K=4) spends the most tokens but gains only 0.6 points over SkillOpt. TextualRL with K=2 scores 1.3 points higher than SkillOpt (K=4) while using 47% fewer tokens. K therefore lets practitioners trade accuracy for cost.

Selection among the three completed runs. As a descriptive comparison, we select the highest test score among the three completed SkillOpt runs separately for each benchmark. This oracle selection favors SkillOpt and gives the following scores.

SearchQASpreadsheetOfficeQADocVQALiveMathALFWorldAvg.
Oracle best of three SkillOpt runs80.3657.1446.5193.3248.8082.0968.04
TextualRL, three-run mean82.6260.2447.0994.0356.0084.0870.68

Even this oracle selection among the observed runs remains below TextualRL's mean on every benchmark. It describes these completed runs, not an upper bound on future restarts. Their individual token costs were not measured for this comparison.

AuthorsOpenReview ↗

General Response to All Reviewers (Part 3/3)

G3. The reinforcement-learning framing (ULn4-W3, hihA-W1)

We have distinguished the reward objective from the procedure used to optimize it (Section 3.1). The frozen target defines a context-conditioned policy, and Eq. (1) seeks a context that maximizes expected task reward:

J(C)=ExEτ∼πθ(⋅∣x,C)[R(τ)].J(C)=\mathbb{E}_{x}\mathbb{E}_{\tau\sim\pi_\theta(\cdot\mid x,C)}[R(\tau)].

TextualRL collects trajectories under the current context CtC_t and proposes bounded textual patches using an optimizer LLM. It accepts a candidate only when its recorded held-out validation score improves. We describe this as gradient-free, reward-driven policy optimization in context space. The validation rule is empirical, rather than a guarantee that expected reward increases. The method does not estimate a policy gradient or provide formal credit-assignment or convergence guarantees.

The correspondence to group-relative rewards. For the binary outcome rewards used in our routing, define the stabilized group-relative advantage as Ai=(ri−mean(r))/(std(r)+ϵ)A_i=(r_i-\mathrm{mean}(\mathbf r))/(\mathrm{std}(\mathbf r)+\epsilon). Mixed groups have nonzero reward advantages, while all-success and all-failure groups have zero reward advantages. TextualRL uses within-task contrasts for mixed groups and cross-task preservation or repair for uniform groups. This correspondence concerns the reward-advantage term in GRPO. It does not imply that other terms, such as KL regularization, contribute no gradient.

Frequency of uniform groups. We counted task groups in the Qwen3.8-27B training records, pooling steps within each benchmark. Each step contains up to 40 tasks, with smaller batches for short datasets and final batches. The Avg. column weights the six benchmarks equally.

SearchQASpreadsheetOfficeQADocVQALiveMathALFWorldAvg.
Mixed12%24%32%10%51%28%26%
All-success71%58%32%83%21%65%55%
All-failure16%18%36%7%29%7%19%
Uniform88%76%68%90%49%72%74%

Uniform groups account for 74% on average across benchmarks. They supply no within-group reward contrast, but their trajectories can still contain evidence about shared successful behaviors or recurring failures. This motivates the cross-task routes without establishing equivalence to GRPO or an empirical advantage over it. LiveMath supplies the largest proportion of mixed groups, whereas DocVQA is dominated by all-success groups.

We have clarified the objective, context update, and validation estimator in Section 3.1, and the scope of the GRPO analogy in Section 3.2. The abstract, introduction, and conclusion now describe TextualRL as gradient-free, reward-driven policy optimization in context space.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.