acceptodds
Under review as a conference paper at ICLR 2027

TITRATE: Adapting Self-Distillation Weight and Skill Refresh to Training Dynamics

19 views

Abstract

Skill-augmented On-Policy Self-distillation trains large language model (LLM) agents by combining Reinforcement Learning (RL) with guidance from a teacher that shares the student policy's parameters but receives hindsight-derived skills as privileged context. However, fixed skill-refresh rules and distillation weights leave skill evolution poorly coordinated with policy learning. We introduce , a unified framework that coordinates these processes through: (1) a that uses an analyzer to consolidate successful and failed rollout evidence at task or class granularity and supports incremental, version-compatible updates; (2) an controller that uses rollout evidence and confidence-based ranking to prioritize skill refreshes under analyzer-call budgets; and (3) a that combines progress-aware cosine annealing with bounded, conflict-aware refinement to balance skill-informed guidance and reward-driven learning. Together, they form a closed co-evolutionary loop: accumulated experience informs skill refinement, skill-informed self-distillation guides policy updates, and the updated policy, in turn, generates new experience. On ALFWorld, WebShop, and SearchQA with three Qwen models (1.7B–7B), TITRATE achieves the highest mean task performance in eight of nine settings, while reducing analyzer calls by 88.8–99.4% and analyzer token consumption by 8.1–56.7% relative to per-trajectory extraction. On class-level SearchQA with QWen2.5-3B, TITRATE ranks second while enabling mixed-outcome updates that restrictive baselines suppress despite unresolved failures. Ablations further support the contributions of evidence-gated refresh and adaptive weighting. Our results demonstrate that coordinating skill evolution and distillation strength with policy learning can achieve a better performance–cost trade-off, supporting TITRATE as an effective solution for self-distillation RL training.

Primary area:
reinforcement learning
Keywords:
Large language model agents, reinforcement learning, on-policy self-distillation, skill evolution, adaptive distillation weighting, budget-aware skill refresh, versioned skill banks

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Akc7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TITRATE trains LLM agents with L=LRL+λtLOPDL = L_{RL} + \lambda_t L_{OPD}, where the teacher is the same model given skills written by an external analyzer (GPT-4.1). It adds a versioned skill bank, a controller that refreshes skills only for persistently failing or stalled keys under a budget, and a two-stage schedule for λt\lambda_t (a progress-aware cosine plus a correction when RL and OPD gradients conflict). On ALFWorld, WebShop, and a Search-R1 QA mixture with three Qwen models, it is best in 8 of 9 settings and uses 88.8% to 99.4% fewer analyzer calls than per-trajectory extraction.

Critical strengths
  1. The evaluation is broad (three benchmarks, three model sizes) and reports analyzer calls and tokens for every method, with a clear analysis of why simple refresh rules fail.
  2. Each component is ablated under two λ\lambda ranges, and four seeds of the full method vary by only 0.80 to 1.45 points.
Critical weaknesses
  1. Margins over the best baseline are small (−0.24 to +2.50 points), about one to three of 128 validation episodes. The reported sd is over checkpoints of one run, and baselines have no repeated runs. The same WebShop setting gives 0.6477 in Table 1 and 0.7000 in Appendix G.2.
  2. There is no RL-only baseline and no comparison with published methods besides an adapted SEED. On SearchQA, TITRATE uses more analyzer tokens than BANK-12 and BANK-FAIL.
  3. The method has many thresholds, and the λ\lambda range must be tuned per benchmark (the higher range makes WebShop collapse to zero).
Questions for authors
  1. How does GRPO without the OPD term perform?
  2. Can you run the strongest bank baseline with several seeds?
  3. Why do Table 1 and Appendix G.2 differ by more than five points on WebShop with Qwen3-1.7B?
  4. Which split and how many questions are used for SearchQA validation?
  5. Can you compare with at least one published skill-based method under your setup?

Suggestions for future work:

  1. Develop an automatic way to choose the λ\lambda range, and rename "SearchQA" to avoid confusion with the existing dataset.
5 comments
AuthorsOpenReview ↗

Response to Reviewer Akc7 (1/3)

We appreciate the reviewer Akc7’s careful reading and the constructive questions. We address each point in turn: comparison with published and reproduced methods and an RL-only baseline (QA1, QA5, CW2), analyzer cost on SearchQA (CW2), the Table 1 vs. Appendix G.2 difference (QA3, CW1), the SearchQA validation split and naming (QA4, suggestion), seed variance of the strongest baselines (QA2, CW1), and the λ range (CW3, suggestion).

1. [QA1 + QA5 + CW2] Main result: comparison with the existing methods

QA1: "How does GRPO without the OPD term perform?"

QA5: "Can you compare with at least one published skill-based method under your setup?"

Conclusion: Among the methods without SFT in Table R1, TITRATE beats GRPO in all 9 settings and Skill-SD in 8 of 9 at matched steps, ranking first in three settings. It does so while building its skill bank from scratch, whereas Skill-SD, SDAR and the other Fixed baselines start from a pre-built o3 skill bank. To isolate skill handling from SFT, we build SEED without SFT (PER-TRAJ): TITRATE outperforms it in all 9 settings with 88.8–99.4% fewer analyzer calls. Against fixed skill-bank rules (BANK-*), TITRATE has the highest success in 8 of 9 settings, and the cheaper rules save cost only by suppressing refreshes. Overall, without SFT or a pre-built skill bank, TITRATE performs strongly at a fraction of the analyzer cost, and none of the simpler refresh rules we test can replace its controller.

Baseline Criteria: We compare policies evaluated without skills, which measures how much the policy itself improves. Therefore, methods whose policy uses skills at inference time (Skill1, SkillRL) are excluded. Methods with offline, pre-built skills (Fixed in Table R1) are included: they use a fixed SkillBank generated by o3, whereas TITRATE starts empty and builds its skills online. OPID and SEED generate skills online, but write new ones for each trajectory.

Evidence: Since rerunning every published method for 300 steps exceeds our compute budget, we reproduce OPID and GiGPO, build the comparison in five steps, each addressing one confound. We emphasize that evaluating at step 150 or 200 of TITRATE’s 300 training steps understates it: its cosine-annealed distillation weight is still mid-schedule, and its success rate is still rising.

Steps 1)–2) compare at matched training steps; step 3) compares our full schedule with SFT-initialized methods; steps 4)–5) are controlled comparisons with our manuscript’s baselines. For completeness, we also include TITRATE with the higher λ range [0.01, 0.1] (Section 4.3.1), omitting WebShop, where this range is unstable (Appendix G.2).

1) Matched steps vs published and reproduced methods (step 150, Table R1). TITRATE is above GRPO and Skill-GRPO in all 9 settings, Skill-SD in 8 of 9, RLSD and GRPO+OPSD in 7 of 9, and our reproduced OPID in 5 of 9. It is competitive with SDAR despite SDAR’s pre-built skills, and ahead of it on 3B and 7B SearchQA (43.8 vs. 43.4; 49.4 vs. 49.0).

Table R1. Success rate (%) at 150 training steps

MethodSkill source1.7B ALF1.7B Web1.7B SQA3B ALF3B Web3B SQA7B ALF7B Web7B SQA
1. RLGRPO—46.138.340.875.063.336.481.272.642.0
Skill-GRPOFixed21.146.140.460.260.934.169.571.940.3
GiGPO (reproduced)—58.853.141.086.362.6—90.872.8—
2. Skill-conditioned RL + OPDOPID (reproduced)Online50.063.341.275.066.445.588.374.248.0
SDARFixed53.958.641.984.468.043.485.982.849.0
RLSDFixed42.250.840.679.766.443.882.077.349.0
GRPO+OPSDFixed32.038.342.281.266.444.680.476.547.0
Skill-SDFixed52.353.940.873.464.044.185.176.547.8
3. TITRATE (step 150 of 300)main configuration: λ [0.0021, 0.0213]Online, selectively refreshed53.155.540.879.767.243.885.277.349.4
higher range: λ [0.01, 0.1]Online, selectively refreshed60.9—41.088.3—43.885.9—49.4

Skill source: Fixed = a read-only skill library from SkillRL (generated by o3), not updated during training; Online = skills written by an analyzer (GPT-4.1) from the method’s own rollouts during training; selectively refreshed = a skill is rewritten only when TITRATE’s controller detects it is stale.

AuthorsOpenReview ↗

Response to Reviewer Akc7 (2/3)

2) Matched steps vs GiGPO on SearchQA (step 200, Table R2). TITRATE is ahead by +1.7 points (3B) and +2.8 points (7B).

Table R2. SearchQA success rate (%) at 200 training steps

Method3B SQA7B SQA
GiGPO42.147.2
TITRATE, main configuration: λ \in [0.0021, 0.0213]43.850.0
Δ (TITRATE − GiGPO)+1.7+2.8

3) Full schedule vs SFT-initialized methods (steps 150 and 300, Table R3). Within 300 steps and without an SFT stage, TITRATE’s best checkpoint matches or exceeds SEED in 7 of 9 settings. SEED’s lead is concentrated at 1.7B ALFWorld and WebShop.

Motivation: SEED starts RL from an SFT checkpoint, so we show how much of its lead TITRATE recovers without SFT over the full schedule.

Table R3. TITRATE vs the SFT-initialized SEED at 150 and 300 steps

Method1.7B ALF1.7B Web1.7B SQA3B ALF3B Web3B SQA7B ALF7B Web7B SQA
1.SFT + RLSEED92.077.342.291.878.945.796.178.148.6
2.TITRATE (no SFT, step 150 of 300)main configuration: λ \in [0.0021, 0.0213]53.155.540.879.767.243.885.277.349.4
higher λ range [0.01, 0.1]60.9—41.088.3—43.885.9—49.4
3.TITRATE (no SFT, step 300)main configuration: λ [0.0021, 0.0213]85.271.947.393.882.547.896.085.552.1
higher λ range [0.01, 0.1]84.6—44.995.3—46.996.3—52.0

4) Controlled comparison vs SEED without SFT (PER-TRAJ, Table 1 in our manuscript). TITRATE outperforms PER-TRAJ in all 9 settings while using 88.8–99.4% fewer analyzer calls and 8.1–56.7% fewer tokens.

Motivation: with SFT removed and the same base checkpoint, PER-TRAJ isolates SEED’s per-trajectory skill extraction, so the performance and cost differences come from skill handling alone.

5) Controlled comparison vs simplified skill banks (BANK-STEP, BANK-12, BANK-FAIL, Table 1 in our manuscript). TITRATE has the highest mean success in 8 of 9 settings and never falls below PER-TRAJ, while each fixed rule does in 3–6 settings. Rules that are cheaper on SearchQA save cost by suppressing refreshes, and they trail TITRATE by 1.2–3.4 points.

Motivation: to test whether a simpler refresh rule could replace TITRATE’s controller.

2. [CW2] Analyzer cost on SearchQA

CW2: "On SearchQA, TITRATE uses more analyzer tokens than BANK-12 and BANK-FAIL."

We agree that on SearchQA, TITRATE uses more analyzer tokens than BANK-12 and BANK-FAIL (160.8–199.9M vs. 11.2–29.7M). But this is a deliberate trade-off rather than inefficiency:

  • Where their savings come from. On class-level SearchQA, their fixed rules skip most refreshes. BANK-12 reuses the old skill on 91.25% of revisits, because keys return before its 12-step threshold. BANK-FAIL blocks refresh on 74.05–74.52% of revisits, because the task group contains both successes and failures. Under these rules, 3–5 of the 16 class skills are never refreshed after creation (Section 4.2).
  • What TITRATE spends the extra tokens on. Its CONTRAST lane admits exactly these mixed-outcome keys once their success rate stalls. Such refreshes make up 88.96–94.88% of TITRATE's SearchQA refreshes.
  • What this buys. TITRATE's SearchQA success is 1.22–3.41 points higher than BANK-12 and BANK-FAIL across the three models.
  • Overall cost. On SearchQA, TITRATE still uses 99.4% fewer analyzer calls and 53.9–56.7% fewer tokens than PER-TRAJ. Across all 9 settings, it uses fewer calls and tokens than BANK-STEP.

We will state this trade-off more clearly in Section 4.2 of the revision.

3. [QA3 + CW1] Table 1 vs. Appendix G.2 difference on WebShop

QA3: "Why do Table 1 and Appendix G.2 differ by more than five points on WebShop with Qwen3-1.7B?"

The performance difference between Table 1 and Appendix G.2 is caused by different training horizons:

  • Table 1 run under the 300 training steps: 64.77 ± 6.14 (4.1 Training configuration)
  • Appendix G.2 runs under the 320 training steps: 70.00 ± 4.47 (G.2 paragraph 2 "The distillation-weight range...")

The 5.23-point gap is comparable to the performance change over 20 training steps, and lies within 1 std unit of the Table 1 result.

AuthorsOpenReview ↗

Response to Reviewer Akc7 (3/3)

4. [QA4 + Suggestion] SearchQA training and validation settings

QA4: "Which split and how many questions are used for SearchQA validation?"

Suggestion: "rename "SearchQA" to avoid confusion with the existing dataset."

  • Training: 169,615 questions from the train splits of NQ (≈47%) and HotpotQA (≈53%), following Search-R1 (Jin et al., 2025).
  • Validation: a fixed 512-question subset of the Search-R1 evaluation pool, sampled per source in proportion to the full pool (table below). NQ and HotpotQA questions come from held-out splits; the other five sources are never used in training.
SourceOrigin splitQuestions in subsetShare of subset (share of full set)Train split used in RL?
PopQAtest14127.5% (27.6%)no
2WikiMultiHopQAdev12524.4% (24.3%)no
TriviaQAtest11221.9% (21.9%)no
HotpotQAdev7314.3% (14.3%)yes
NQtest367.0% (7.0%)yes
MuSiQuedev244.7% (4.7%)no
Bamboogletest10.2% (0.2%)no
Total512

Naming: we will rename the benchmark to "Search-R1 QA" to avoid confusion with the existing SearchQA dataset.

5. [QA2 + CW1] Seed variance of the strongest baselines

QA2: "Can you run the strongest bank baseline with several seeds?"

CW1: "baselines have no repeated runs."

Across three seeds, TITRATE's reported success remains above the strongest baseline's seed mean on ALFWorld, WebShop and SearchQA. On ALFWorld, TITRATE is also 6–14× more stable across seeds than the baseline.

  1. Experiments: Due to the compute budget constraints, we reran the strongest bank baseline for each Qwen3-1.7B benchmark with 2 additional seeds in 300 training steps.

Table R4. BANK-FAIL across three seeds on ALFWorld / Qwen3-1.7B.

SeedSuccess Rate ↑RefreshAnalysis Call ↓
0 (paper Table 1)0.7617 ± 0.05098653,559
10.6977 ± 0.06468563,539
20.7953 ± 0.05218353,540
across-seed mean ± sd0.7516 ± 0.0496852 ± 153,546 ± 11
across-seed cv6.60%1.81%0.32%

Table R5. BANK-FAIL across three seeds on WebShop / Qwen3-1.7B.

SeedSuccess Rate ↑RefreshAnalysis Call ↓
0 (paper Table 1)0.6256 ± 0.04322694,481
10.6086 ± 0.04652514,456
20.6641 ± 0.03612984,487
across-seed mean ± sd0.6328 ± 0.0284273 ± 244,475 ± 16
across-seed cv4.49%8.70%0.37%

Table R6. BANK-STEP across three seeds on SearchQA / Qwen3-1.7B.

SeedSuccess Rate ↑RefreshAnalysis Call ↓
0 (paper Table 1)0.4199 ± 0.01964,2854,301
10.3992 ± 0.01934,2974,313
20.4279 ± 0.01214,2774,312
across-seed mean ± sd0.4157 ± 0.01484,286 ± 104,309 ± 7
across-seed cv3.56%0.23%0.15%

2) Observation 1: margins (Tables R4–R6). TITRATE exceeds the baseline’s three-seed mean by +3.51 pp on ALFWorld, +1.49 pp on WebShop and +1.35 pp on SearchQA.

Implication: these gains are comparable to seed variation, so we will report three-seed means in the revision.

3) Observation 2: stability on ALFWorld (Table R4; Table 18, Section G.1). BANK-FAIL’s across-seed cv in success is 6.60%, versus 0.46% / 1.06% for TITRATE in the higher / lower λ ranges, while TITRATE’s refresh count varies more (cv 5.7% / 7.4% vs. 1.8%).

Implication: TITRATE adapts its refreshes to each seed, yet reaches nearly the same final success.

6. [CW3] Choosing the λ range

CW3: "the λ range must be tuned per benchmark (the higher range makes WebShop collapse to zero)."

We agree that setting λt\lambda_t automatically is a valuable direction, but it requires careful design. For example, instead of a fixed range, λt\lambda_t could start from a single value λ0\lambda_0 and be adjusted by TITRATE’s own training signals (progress vtv_t and RL–OPD conflict qtq_t), with a stop signal that halts increases before distillation becomes excessive. We will add this discussion to “Further Study for λt\lambda_t and Schedule” and leave the design to future work.

Reviewer_MofkOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper tries to improve skill augmented on policy distillation by a versioned skill bank and budget aware gating. But it is unclear how exactly it is done and what is the significant results.

Critical strengths

I really couldn't follow this paper. No strength found.

Critical weaknesses
  • The results section looks extremely weak, it hasn't compared itself with OPID or SDAR that is the basis of the work. By basis I refer to "Skill-augmented On-Policy Self-distillation trains large language model (LLM) agents by combining Reinforcement Learning (RL) with guidance from a teacher that shares the student policy’s parameters but receives hindsight-derived skills as privileged context. However, fixed skill-refresh rules and distillation weights leave skill evolution poorly coordinated with policy learning"

  • The figures looks like AI generated by modifying OPID's paper figures.

  • The text is extremely hard to follow.

Questions for authors

Could you please explain your paper in simpler high level terms?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Mofk (1/3)

We thank the reviewer Mofk and address each point: a plain-language summary of TITRATE (QA1, CW3), comparison with OPID/SDAR and other published methods (CW1), and the overview figure (CW2).

1. [QA1 + CW3] A simplified explanation

QA1: "Could you please explain your paper in simpler high level terms?"

CW3: "The text is extremely hard to follow."

Skill-augmented agents learn from two signals: the task reward (RL), and on-policy distillation (OPD) from a “self-teacher”: the same model, given short written skills summarized from past attempts. TITRATE makes two decisions adaptive in this process:

(i) When to rewrite a skill. Writing skills requires calls to an analyzer model, which is costly. Prior methods either regenerate skills for every trajectory or rewrite them on a fixed schedule. TITRATE reuses a skill until the evidence shows it is stale: its corresponding task keeps failing in every attempt, or its success rate has stopped improving.

(ii) How much to trust the teacher. The teacher helps most early in training, but later its guidance may conflict with the reward. TITRATE lowers the teacher’s weight along a cosine schedule, with two adjustments: the schedule moves faster or slower depending on how quickly success is improving, and the weight is lowered further (within a bound) when the RL and OPD updates conflict.

Conclusion: With these designs, TITRATE stays competitive at matched training steps with the methods without SFT in Table R1 (beating GRPO in all 9 settings and Skill-SD in 8 of 9, first in three settings). It achieves this while building its skill bank from scratch, whereas Skill-SD, SDAR and the other Fixed baselines start from a pre-built o3 skill bank. To isolate skill handling from SFT, we build SEED without SFT (PER-TRAJ). TITRATE outperforms it in all 9 settings with 88.8–99.4% fewer analyzer calls. We also find that no fixed skill-bank refresh rule matches both its success and its analyzer cost across settings. Together, these results show that TITRATE’s two designs, selective skill refresh and adaptive teacher weighting, make it competitive without SFT or a pre-built skill bank, at a fraction of the analyzer cost.

AuthorsOpenReview ↗

Response to Reviewer Mofk (2/3)

2. [CW1] Main result: comparison with the existing methods

CW1: "it hasn't compared itself with OPID or SDAR that is the basis of the work."

Baseline Criteria: We compare policies evaluated without skills, which measures how much the policy itself improves. Therefore, methods whose policy uses skills at inference time (Skill1, SkillRL) are excluded. Methods with offline, pre-built skills (Fixed in Table R1) are included: they use a fixed SkillBank generated by o3, whereas TITRATE starts empty and builds its skills online. OPID and SEED generate skills online, but write new ones for each trajectory.

Evidence: Since rerunning every published method for 300 steps exceeds our compute budget, we reproduce OPID and GiGPO, build the comparison in five steps, each addressing one confound. We emphasize that evaluating at step 150 or 200 of TITRATE’s 300 training steps understates it: its cosine-annealed distillation weight is still mid-schedule, and its success rate is still rising.

Steps 1)–2) compare at matched training steps; step 3) compares our full schedule with SFT-initialized methods; steps 4)–5) are controlled comparisons with our manuscript’s baselines. For completeness, we also include TITRATE with the higher λ range [0.01, 0.1] (Section 4.3.1), omitting WebShop, where this range is unstable (Appendix G.2).

1) Matched steps vs published and reproduced methods (step 150, Table R1). TITRATE is above GRPO and Skill-GRPO in all 9 settings, Skill-SD in 8 of 9, RLSD and GRPO+OPSD in 7 of 9, and our reproduced OPID in 5 of 9. It is competitive with SDAR despite SDAR’s pre-built skills, and ahead of it on 3B and 7B SearchQA (43.8 vs. 43.4; 49.4 vs. 49.0).

Table R1. Success rate (%) at 150 training steps

MethodSkill source1.7B ALF1.7B Web1.7B SQA3B ALF3B Web3B SQA7B ALF7B Web7B SQA
1. RLGRPO—46.138.340.875.063.336.481.272.642.0
Skill-GRPOFixed21.146.140.460.260.934.169.571.940.3
GiGPO (reproduced)—58.853.141.086.362.6—90.872.8—
2. Skill-conditioned RL + OPDOPID (reproduced)Online50.063.341.275.066.445.588.374.248.0
SDARFixed53.958.641.984.468.043.485.982.849.0
RLSDFixed42.250.840.679.766.443.882.077.349.0
GRPO+OPSDFixed32.038.342.281.266.444.680.476.547.0
Skill-SDFixed52.353.940.873.464.044.185.176.547.8
3. TITRATE (step 150 of 300)main configuration: λ [0.0021, 0.0213]Online, selectively refreshed53.155.540.879.767.243.885.277.349.4
higher range: λ [0.01, 0.1]Online, selectively refreshed60.9—41.088.3—43.885.9—49.4

Skill source: Fixed = a read-only skill library from SkillRL (generated by o3), not updated during training; Online = skills written by an analyzer (GPT-4.1) from the method’s own rollouts during training; selectively refreshed = a skill is rewritten only when TITRATE’s controller detects it is stale.

2) Matched steps vs GiGPO on SearchQA (step 200, Table R2). TITRATE is ahead by +1.7 points (3B) and +2.8 points (7B).

Table R2. SearchQA success rate (%) at 200 training steps

Method3B SQA7B SQA
GiGPO42.147.2
TITRATE, main configuration: λ \in [0.0021, 0.0213]43.850.0
Δ (TITRATE − GiGPO)+1.7+2.8
AuthorsOpenReview ↗

Response to Reviewer Mofk (3/3)

3) Full schedule vs SFT-initialized methods (steps 150 and 300, Table R3). Within 300 steps and without an SFT stage, TITRATE’s best checkpoint matches or exceeds SEED in 7 of 9 settings. SEED’s lead is concentrated at 1.7B ALFWorld and WebShop.

Motivation: SEED starts RL from an SFT checkpoint, so we show how much of its lead TITRATE recovers without SFT over the full schedule.

Table R3. TITRATE vs the SFT-initialized SEED at 150 and 300 steps

Method1.7B ALF1.7B Web1.7B SQA3B ALF3B Web3B SQA7B ALF7B Web7B SQA
1.SFT + RLSEED92.077.342.291.878.945.796.178.148.6
2.TITRATE (no SFT, step 150 of 300)main configuration: λ \in [0.0021, 0.0213]53.155.540.879.767.243.885.277.349.4
higher λ range [0.01, 0.1]60.9—41.088.3—43.885.9—49.4
3.TITRATE (no SFT, step 300)main configuration: λ [0.0021, 0.0213]85.271.947.393.882.547.896.085.552.1
higher λ range [0.01, 0.1]84.6—44.995.3—46.996.3—52.0

4) Controlled comparison vs SEED without SFT (PER-TRAJ, Table 1 in our manuscript). TITRATE outperforms PER-TRAJ in all 9 settings while using 88.8–99.4% fewer analyzer calls and 8.1–56.7% fewer tokens.

Motivation: with SFT removed and the same base checkpoint, PER-TRAJ isolates SEED’s per-trajectory skill extraction, so the performance and cost differences come from skill handling alone.

5) Controlled comparison vs simplified skill banks (BANK-STEP, BANK-12, BANK-FAIL, Table 1 in our manuscript). TITRATE has the highest mean success in 8 of 9 settings and never falls below PER-TRAJ, while each fixed rule does in 3–6 settings. Rules that are cheaper on SearchQA save cost by suppressing refreshes, and they trail TITRATE by 1.2–3.4 points.

Motivation: to test whether a simpler refresh rule could replace TITRATE’s controller.

3. Ablations show where TITRATE’s performance comes from:

  • Controller design. Separate FAILURE and CONTRAST lanes refresh only the skills that need it, which keeps refresh cost low.
  • Two-stage training schedule. The OPD weight adapts to training progress and to RL–OPD conflict in the gradient update.

4. Further analyses:

  • Analyzer choice (Section G.3): how different analyzer models affect final success, which supports our choice of analyzer.
  • Skill key granularity (Section E.2): how class-level keys affect extraction cost and task performance, complementing the main results.
  • Consistency across seeds (Section G.1): TITRATE’s success is more stable across seeds than BANK-FAIL’s.
  • Training speed (Section E.3.3): TITRATE reaches the same success rate in fewer steps, with the size of the speed-up varying by model.

5. [CW2] Overview figure and its relation to OPID

CW2: "The figures looks like AI generated by modifying OPID's paper figures."

Figure 1 is not derived from OPID’s figure (arXiv:2606.26790, page 4); the two figures depict different methods. As disclosed in our AI use statement, we designed Figure 1, and generative AI tools were used to refine its visual design. Figure 1 shows three components that OPID does not have: a versioned skill bank, selective refresh, and a distillation weight that adapts to learning progress and estimated RL–OPD conflict. These are our central contributions. In the revision, we will redraw Figure 1 in the same style as Figure 2, with larger fonts and the three components labelled explicitly.

Reviewer_56FqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

TITRATE studies how skill updates and policy learning can be coordinated in skill-augmented OPSD. When student is updated with a joint objective combining RL and skill-conditioned OPD, the REUSE/REFRESH controller chooses between reuse and refresh based on current outcomes and historical records. Keys that fail persistently (all rollouts fail) and whose rollouts in the previous serving have nearly identical mean advantages enter the FAILURE lane; mixed-outcome keys that still have sufficient failure evidence but whose success rate has stalled enter the CONTRAST lane. The distillation weight is determined in two stages: first, the progress clock of a cosine annealing schedule is advanced according to changes in training success rates; second, the current-step weight receives a bounded downward correction based on the current RL–OPD conflict proxy and an optimizer-history estimate.

Experiments cover ALFWorld, WebShop, and SearchQA with three Qwen models ranging from 1.7B to 7B. TITRATE achieves the highest mean task performance in eight of the nine settings; relative to PER-TRAJ, it uses 88.8–99.4% fewer analyzer calls and 8.1–56.7% fewer analyzer tokens. Ablations on ALFWorld/1.7B tighten the FAILURE and CONTRAST refresh conditions, replace the controller with all-failure refresh, change cosine to linear, and disable Stage 1 or Stage 2. Additional analyses further examine key granularity, actual refresh behavior, the weight range, and analyzer choice.

Critical strengths
  • Strength 1: The refresh rules distinguish failure regimes and address a concrete issue in class-level skill sharing. The SearchQA analysis documents cross-task suppression under BANK-FAIL, while most TITRATE refreshes are actually executed through the CONTRAST pathway

  • Strength 2: Using two-stage schedule to separate learning progress from current teacher interaction. This separation allows the longer-term schedule and the immediate correction to use their respective information, while explicitly bounding the influence of the current estimates on the coefficient.

Critical weaknesses
  • Weakness 1: FAILURE admission has a persistent blind spot that keeps it from refreshing. Equation (3) requires all preceding prompt groups to be advantage-flat. Consequently, even if the task fails continuously and completely, the key cannot refresh — and cannot enter the CONTRAST lane — as long as factors such as invalid-action penalties cause the advantages of failed trajectories to differ. The manuscript provides no fallback mechanism for prolonged failure. The existence of reward differences among failed trajectories does not imply that the task is making progress, and it is can't to justify reusing the old skill.

  • Weakness 2: The budget mechanism is decoupled from the main cost, and its contribution to ordering is never tested in the main experiments. In the main experiments (βFAILURE=1\beta_{\mathrm{FAILURE}}=1), the admitted keys in both lanes all refresh, so ordering does not change the selection outcome; and the CONTRAST lane, which accounts for roughly 89%–95% of refreshes on SearchQA, is not subject to the FAILURE quota. In the actual results, relative to BANK-FAIL, TITRATE uses 11.6–17.8× more analyzer tokens while improving by only 1.22–2.42 percentage points. This pronounced cost disadvantage directly weakens the paper's performance–cost trade-off contribution.

Questions for authors
  • Question 1: How many keys that fail completely and continuously are repeatedly denied refresh because of non-flat advantages? Do these keys eventually refresh successfully? Why can prolonged failure not override this veto condition?

  • Quesion 2: Adjacent servings in SearchQA contain different questions. How does Equation (4) distinguish changes in task-difficulty composition from genuine learning stagnation? The current Wilson bound constrains the failure rate; it does not address this confounder in cross-batch success-rate comparisons.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 56Fq (1/3)

We appreciate reviewer 56Fq’s careful reading of our controller design. We address each point: the FAILURE-lane veto under non-flat advantages (CW1, QA1), question-difficulty shifts in the CONTRAST condition on SearchQA (QA2), the budget mechanism and analyzer cost relative to BANK-FAIL (CW2).

1. [CW1 + QA1] FAILURE-lane veto under non-flat advantages

CW1: "The manuscript provides no fallback mechanism for prolonged failure."

QA1: "Why can prolonged failure not override this veto condition?"

Following the reviewer's term, we define a veto as a zero-success serving whose previous serving also had zero successes but was not advantage-flat. Vetoes are rare and short-lived: they affect at most 4.5% of rollouts and last at most 3 consecutive servings for task keys (7–15 for class keys; Table R1).

Table R1. How often veto on FAILURE-lane occurs.

SettingSourceAdmittedVetoVeto / revisitsVeto / rolloutsMax consecutive vetoes
ALFWorld (task)TITRATE, 6 runs7902862.3%1.0%3
ALFWorld (class)TITRATE, 6 runs2402012.0%2.4%12
WebShop (task)TITRATE, 3 runs14522312.7%1.5%3
WebShop (class)TITRATE, 3 runs1063145.7%4.5%15
SearchQA (class)Replay, 12 baseline runs1,6791480.3%0.1%7

1) Why advantage-flatness. When every rollout in a group fails with identical advantages, group-normalized RL gets no gradient. For the FAILURE lane, a refreshed skill changes the teacher's privileged context and thus the OPD loss, so the student keeps learning. Failed groups with non-flat advantages still carry an RL gradient, so we reuse the current skill.

2) Why prolonged failure does not override the veto. Such an override fits our design, but we omitted it for two reasons. First, vetoes are short-lived (Table R1), so it would rarely trigger. Second, failure duration alone does not show whether a refresh will help: in one ALFWorld class run, a key stayed at zero success for 147 servings despite 74 refreshes.

3) Limitation. We agree that this proxy is imperfect: both the reward (e.g., partial scores) and reward shaping (e.g., invalid-action penalties) design can make failed rollouts’ advantages non-flat without success progress. We will explore (i) the reviewer's fallback for prolonged failure: admitting a key after K consecutive all-failure servings, with a cap on unproductive refresh; and (ii) computing advantage-flatness on the task-success signal to avoid irrelevant reward and reward shaping signals.

AuthorsOpenReview ↗

Response to Reviewer 56Fq (2/3)

2. [QA2] Question-difficulty shifts in the CONTRAST condition on SearchQA

QA2: "How does Equation (4) distinguish changes in task-difficulty composition from genuine learning stagnation?"

Eq. (4) does not explicitly separate difficulty shifts from stagnation. We show (1) that It relies on key granularity and rarely changes refresh decisions, and (2) why the Wilson bound applies only to the current serving.

1) How Eq. (4) handles question mix.

(a) By construction. Task-level keys (ALFWorld, WebShop) repeat the same task at every serving, so question mix cannot change. On class-level SearchQA keys, a shift toward harder questions can look like stagnation. Yet, refresh also requires mixed outcomes and Ek(t)>τUE_k(t) > \tau_{\mathcal U}, and admitted keys are ranked by Ek(t)E_k(t), so it only targets skills with more remaining failure.

(b) Empirically, the confounder barely changes decisions. All SearchQA runs share one question stream, so we estimate each question’s difficulty from the other runs. Question mix does move success rates (predicted vs. observed change: r = 0.79). Even so, a difficulty-adjusted Eq. (4) changes total CONTRAST refreshes by only 2%, and its per-key refresh counts correlate with the raw rule at r = 0.98.

2) Why the Wilson bound applies only to the current serving. Eq. (4) refreshes a key only when two conditions hold: no improvement over the previous serving (I: pk(0)≤pk(−1)p_k^{(0)} \le p_k^{(-1)}), and enough failure left in the current serving (W: Ek(t)>τUE_k(t) > \tau_{\mathcal U}). We compare this with a symmetric W that also checks the previous serving.

Table R2. Role of the two CONTRAST conditions, as % of mixed-outcome revisits. “Blocked only by I / W”: revisits that fail only that condition. Last column: share of current refreshes a symmetric W would remove. SearchQA: replay over 12 class-key runs.

Setting (runs)Mixed-outcome revisitsRefreshed (W ∧ I)Blocked only by IBlocked only by WSymmetric W: refreshes removed
ALFWorld task (TITRATE, 6)4,32426.7%43.4%12.9%39.9%
ALFWorld class (TITRATE, 6)6,83147.7%29.1%10.7%46.9%
WebShop task (TITRATE, 3)59921.0%43.9%10.7%31.0%
WebShop class (TITRATE, 3)3,34747.2%35.6%7.1%38.6%
SearchQA class (TITRATE & replay, 12)43,20547.9%48.9%0.7%10.4%

(a) By definition, W measures failure remaining under the current skill. The previous serving came from an earlier policy, so applying τU\tau_{\mathcal U} to it needs a time-dependent discount (like reward discount factor), adding another hyperparameter.

(b) Empirically (Table R2), a symmetric W would block refreshes for keys that regress from a nearly solved state: it removes 31–47% of current CONTRAST refreshes on ALFWorld and WebShop (10% on SearchQA), and about half of those follow a fully solved previous serving.

Given 1) and 2), and to keep Eq. (4) simple, we add no difficulty correction and apply W only to the current serving.

AuthorsOpenReview ↗

Response to Reviewer 56Fq (3/3)

3. [CW2] Budget mechanism and analyzer cost relative to BANK-FAIL

CW2: "relative to BANK-FAIL, TITRATE uses 11.6–17.8× more analyzer tokens while improving by only 1.22–2.42 percentage points."

We agree that ordering is inactive in the main runs. Tightening either knob makes ordering decide selection, but it saves only 4–9% of analyzer calls and lowers success by 1.9–5.1 pp (Table R3), so the main runs keep βFAILURE=1\beta_{\text{FAILURE}} = 1 and τU=0.05\tau_{\mathcal U} = 0.05. TITRATE’s extra cost over BANK-FAIL on SearchQA is a trade-off: it pays for mixed-outcome refreshes that BANK-FAIL never triggers.

1) Testing the knobs (Table R3, ALFWorld). TI-BF halves the FAILURE quota (βFAILURE\beta_{\text{FAILURE}}: 1 → 0.5), and TI-TC raises the CONTRAST threshold (τU\tau_{\mathcal U}: 0.05 → 0.2152). Either cuts refreshes by 19–38% but analyzer calls by only 4–9%, because most calls (≈84%) create skills for newly seen keys. The success drop exceeds the across-seed spread (0.80 / 1.45 pp).

Table R3. Ablation of the budget knobs. (Refresh and call changes are relative to full TITRATE.)

Variant (knob)Higher λ range: successHigher λ range: refreshes / callsLower λ range: successLower λ range: refreshes / calls
TITRATE (full): β_FAILURE = 1, τ_U = 0.050.8077 ± 0.0170533 / 3,3200.6852 ± 0.0466849 / 3,636
TI-BF: β_FAILURE 1.0 → 0.50.7602 ± 0.0244 (−4.75 pp)397 (−25.5%) / 3,184 (−4.1%)0.6344 ± 0.0632 (−5.08 pp)529 (−37.7%) / 3,316 (−8.8%)
TI-TC: τ_U 0.05 → 0.21520.7891 ± 0.0321 (−1.86 pp)404 (−24.2%) / 3,191 (−3.9%)0.6375 ± 0.0518 (−4.77 pp)687 (−19.1%) / 3,474 (−4.5%)

2) Cost relative to BANK-FAIL. BANK-FAIL is cheap on SearchQA because it blocks refresh on 74% of revisits, leaving 3–5 of the 16 class skills never refreshed after creation (Section 4.2). We agree that the resulting gain (+1.22–2.42 pp) is modest for this cost. In the revision, we will anchor TITRATE’s cost claim on per-trajectory extraction (88.8–99.4% fewer calls) and BANK-STEP (fewer calls and tokens in all 9 settings), and present BANK-12/BANK-FAIL as a cost–robustness trade-off: they are cheaper on SearchQA, but neither matches TITRATE across settings.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response Summary — Submission 48299 (TITRATE)

We thank all three reviewers for their feedback, and especially those who provided constructive suggestions. During the response period, we added new baseline comparisons and experiments, and clarified the controller and the evaluation setup:

1. New baseline comparisons: effectiveness of TITRATE

  • Published skill-based and RL methods at matched steps (Mofk CW1; Akc7 QA1, QA5). At step 150 of 300, TITRATE beats GRPO and Skill-GRPO in all 9 settings, Skill-SD in 8 of 9, RLSD and GRPO+OPSD in 7 of 9, and OPID under our setup in 5 of 9. It is competitive with SDAR and outperforms it on 3B/7B SearchQA. TITRATE builds its skill bank from scratch, whereas SDAR, Skill-SD and the other Fixed baselines start from a pre-built o3 skill bank. Methods that learn skills online or use no skill bank (OPID, SEED without SFT, GiGPO) match our setting more closely and are more direct counterparts for fair comparison.
  • GiGPO on SearchQA (step 200). TITRATE is ahead by +1.7 (3B) and +2.8 points (7B).
  • SFT-initialized SEED. Without an SFT stage, TITRATE's best checkpoint within 300 steps matches or exceeds SEED in 7 of 9 settings.
  • SEED without SFT (PER-TRAJ). TITRATE outperforms it in all 9 settings with 88.8–99.4% fewer analyzer calls.
  • Multi-seed baselines (Akc7 QA2, CW1). With two more seeds for the strongest bank baseline on each Qwen3-1.7B benchmark, TITRATE stays above the three-seed mean (+3.51 / +1.49 / +1.35 pp on ALFWorld / WebShop / SearchQA) and is 6–14× more stable across seeds on ALFWorld.
  • Budget-knob ablation (56Fq CW2). Halving the FAILURE quota or raising the CONTRAST threshold cuts refreshes by 19–38% but analyzer calls by only 4–9%, and lowers success by 1.9–5.1 pp.

2. Clarifications

  • Method (Mofk QA1, CW3). TITRATE makes two decisions adaptive: when to rewrite a skill (only when evidence shows it is stale) and how much to trust the self-teacher (a cosine schedule adjusted by learning progress and RL–OPD conflict).
  • Baseline criteria (Mofk CW1; Akc7 QA5). We evaluate policies without skills at inference time, to measure what the policy internalizes. Methods that use skills at inference (Skill1, SkillRL, and similar variations) are therefore excluded.
  • FAILURE-lane veto (56Fq CW1, QA1). Vetoes are rare and short-lived: at most 4.5% of rollouts and 3 consecutive servings for task keys.
  • CONTRAST and question difficulty (56Fq QA2). A difficulty-adjusted Eq. (4) changes CONTRAST refreshes by only 2% (per-key r = 0.98); a symmetric Wilson condition would remove 31–47% of refreshes on ALFWorld and WebShop.
  • Analyzer cost on SearchQA (Akc7 CW2; 56Fq CW2). BANK-12 and BANK-FAIL are cheaper because they skip most refreshes, leaving 3–5 of 16 class skills never refreshed. TITRATE spends the extra tokens on mixed-outcome refreshes and gains 1.22–3.41 points.
  • Table 1 vs. Appendix G.2 (Akc7 QA3). The gap comes from different training horizons (300 vs. 320 steps) and lies within 1 std of Table 1.
  • SearchQA validation (Akc7 QA4). A fixed 512-question subset of the Search-R1 evaluation pool, sampled in proportion to it; five of its seven sources are never used in training.
  • Figure 1 (Mofk CW2). Figure 1 is not derived from OPID's figure. Our contributions are three components that differ from OPID and the other baselines: a versioned skill bank, selective refresh, and an adaptive distillation weight.

3. Key findings

  • TITRATE is competitive with published skill-based methods at matched steps, without SFT or a pre-built skill bank.
  • Success rate is not the full picture: analyzer cost and whether stale skills get refreshed also matter. On all three, TITRATE is the most consistent choice: it beats per-trajectory extraction at a fraction of the analyzer cost, uses fewer calls and tokens than BANK-STEP in all 9 settings, while no fixed refresh rule (BANK-*) does well everywhere.

4. Revision commitments

  • Rename "SearchQA" to "Search-R1 QA" to avoid confusion with the existing dataset.
  • Report three-seed means for the bank baselines.
  • Section 4.2: anchor the cost claim on PER-TRAJ and BANK-STEP, and present BANK-12/BANK-FAIL as a cost–robustness trade-off.
  • Redraw Figure 1 in the style of Figure 2, with larger fonts and the three components labeled.
  • Discuss setting λtλ_t automatically in "Further Study for λtλ_t and Schedule".
  • Limitations: a fallback for prolonged failure (with a cap on unproductive refreshes), and advantage-flatness computed on the task-success signal.

We are grateful for the reviewers' engagement and will continue to improve TITRATE performance and incorporate all feedback in our next revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.