- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
TITRATE trains LLM agents with , where the teacher is the same model given skills written by an external analyzer (GPT-4.1). It adds a versioned skill bank, a controller that refreshes skills only for persistently failing or stalled keys under a budget, and a two-stage schedule for (a progress-aware cosine plus a correction when RL and OPD gradients conflict). On ALFWorld, WebShop, and a Search-R1 QA mixture with three Qwen models, it is best in 8 of 9 settings and uses 88.8% to 99.4% fewer analyzer calls than per-trajectory extraction.
Critical strengths
- The evaluation is broad (three benchmarks, three model sizes) and reports analyzer calls and tokens for every method, with a clear analysis of why simple refresh rules fail.
- Each component is ablated under two ranges, and four seeds of the full method vary by only 0.80 to 1.45 points.
Critical weaknesses
- Margins over the best baseline are small (−0.24 to +2.50 points), about one to three of 128 validation episodes. The reported sd is over checkpoints of one run, and baselines have no repeated runs. The same WebShop setting gives 0.6477 in Table 1 and 0.7000 in Appendix G.2.
- There is no RL-only baseline and no comparison with published methods besides an adapted SEED. On SearchQA, TITRATE uses more analyzer tokens than BANK-12 and BANK-FAIL.
- The method has many thresholds, and the range must be tuned per benchmark (the higher range makes WebShop collapse to zero).
Questions for authors
- How does GRPO without the OPD term perform?
- Can you run the strongest bank baseline with several seeds?
- Why do Table 1 and Appendix G.2 differ by more than five points on WebShop with Qwen3-1.7B?
- Which split and how many questions are used for SearchQA validation?
- Can you compare with at least one published skill-based method under your setup?
Suggestions for future work:
- Develop an automatic way to choose the range, and rename "SearchQA" to avoid confusion with the existing dataset.
5 comments
Response to Reviewer Akc7 (2/3)
2) Matched steps vs GiGPO on SearchQA (step 200, Table R2). TITRATE is ahead by +1.7 points (3B) and +2.8 points (7B).
Table R2. SearchQA success rate (%) at 200 training steps
| Method | 3B SQA | 7B SQA |
|---|---|---|
| GiGPO | 42.1 | 47.2 |
| TITRATE, main configuration: λ \in [0.0021, 0.0213] | 43.8 | 50.0 |
| Δ (TITRATE − GiGPO) | +1.7 | +2.8 |
3) Full schedule vs SFT-initialized methods (steps 150 and 300, Table R3). Within 300 steps and without an SFT stage, TITRATE’s best checkpoint matches or exceeds SEED in 7 of 9 settings. SEED’s lead is concentrated at 1.7B ALFWorld and WebShop.
Motivation: SEED starts RL from an SFT checkpoint, so we show how much of its lead TITRATE recovers without SFT over the full schedule.
Table R3. TITRATE vs the SFT-initialized SEED at 150 and 300 steps
| Method | 1.7B ALF | 1.7B Web | 1.7B SQA | 3B ALF | 3B Web | 3B SQA | 7B ALF | 7B Web | 7B SQA | |
|---|---|---|---|---|---|---|---|---|---|---|
| 1.SFT + RL | SEED | 92.0 | 77.3 | 42.2 | 91.8 | 78.9 | 45.7 | 96.1 | 78.1 | 48.6 |
| 2.TITRATE (no SFT, step 150 of 300) | main configuration: λ \in [0.0021, 0.0213] | 53.1 | 55.5 | 40.8 | 79.7 | 67.2 | 43.8 | 85.2 | 77.3 | 49.4 |
| higher λ range [0.01, 0.1] | 60.9 | — | 41.0 | 88.3 | — | 43.8 | 85.9 | — | 49.4 | |
| 3.TITRATE (no SFT, step 300) | main configuration: λ [0.0021, 0.0213] | 85.2 | 71.9 | 47.3 | 93.8 | 82.5 | 47.8 | 96.0 | 85.5 | 52.1 |
| higher λ range [0.01, 0.1] | 84.6 | — | 44.9 | 95.3 | — | 46.9 | 96.3 | — | 52.0 |
4) Controlled comparison vs SEED without SFT (PER-TRAJ, Table 1 in our manuscript). TITRATE outperforms PER-TRAJ in all 9 settings while using 88.8–99.4% fewer analyzer calls and 8.1–56.7% fewer tokens.
Motivation: with SFT removed and the same base checkpoint, PER-TRAJ isolates SEED’s per-trajectory skill extraction, so the performance and cost differences come from skill handling alone.
5) Controlled comparison vs simplified skill banks (BANK-STEP, BANK-12, BANK-FAIL, Table 1 in our manuscript). TITRATE has the highest mean success in 8 of 9 settings and never falls below PER-TRAJ, while each fixed rule does in 3–6 settings. Rules that are cheaper on SearchQA save cost by suppressing refreshes, and they trail TITRATE by 1.2–3.4 points.
Motivation: to test whether a simpler refresh rule could replace TITRATE’s controller.
2. [CW2] Analyzer cost on SearchQA
CW2: "On SearchQA, TITRATE uses more analyzer tokens than BANK-12 and BANK-FAIL."
We agree that on SearchQA, TITRATE uses more analyzer tokens than BANK-12 and BANK-FAIL (160.8–199.9M vs. 11.2–29.7M). But this is a deliberate trade-off rather than inefficiency:
- Where their savings come from. On class-level SearchQA, their fixed rules skip most refreshes. BANK-12 reuses the old skill on 91.25% of revisits, because keys return before its 12-step threshold. BANK-FAIL blocks refresh on 74.05–74.52% of revisits, because the task group contains both successes and failures. Under these rules, 3–5 of the 16 class skills are never refreshed after creation (Section 4.2).
- What TITRATE spends the extra tokens on. Its CONTRAST lane admits exactly these mixed-outcome keys once their success rate stalls. Such refreshes make up 88.96–94.88% of TITRATE's SearchQA refreshes.
- What this buys. TITRATE's SearchQA success is 1.22–3.41 points higher than BANK-12 and BANK-FAIL across the three models.
- Overall cost. On SearchQA, TITRATE still uses 99.4% fewer analyzer calls and 53.9–56.7% fewer tokens than PER-TRAJ. Across all 9 settings, it uses fewer calls and tokens than BANK-STEP.
We will state this trade-off more clearly in Section 4.2 of the revision.
3. [QA3 + CW1] Table 1 vs. Appendix G.2 difference on WebShop
QA3: "Why do Table 1 and Appendix G.2 differ by more than five points on WebShop with Qwen3-1.7B?"
The performance difference between Table 1 and Appendix G.2 is caused by different training horizons:
- Table 1 run under the 300 training steps: 64.77 ± 6.14 (4.1 Training configuration)
- Appendix G.2 runs under the 320 training steps: 70.00 ± 4.47 (G.2 paragraph 2 "The distillation-weight range...")
The 5.23-point gap is comparable to the performance change over 20 training steps, and lies within 1 std unit of the Table 1 result.
Response to Reviewer Akc7 (3/3)
4. [QA4 + Suggestion] SearchQA training and validation settings
QA4: "Which split and how many questions are used for SearchQA validation?"
Suggestion: "rename "SearchQA" to avoid confusion with the existing dataset."
- Training: 169,615 questions from the train splits of NQ (≈47%) and HotpotQA (≈53%), following Search-R1 (Jin et al., 2025).
- Validation: a fixed 512-question subset of the Search-R1 evaluation pool, sampled per source in proportion to the full pool (table below). NQ and HotpotQA questions come from held-out splits; the other five sources are never used in training.
| Source | Origin split | Questions in subset | Share of subset (share of full set) | Train split used in RL? |
|---|---|---|---|---|
| PopQA | test | 141 | 27.5% (27.6%) | no |
| 2WikiMultiHopQA | dev | 125 | 24.4% (24.3%) | no |
| TriviaQA | test | 112 | 21.9% (21.9%) | no |
| HotpotQA | dev | 73 | 14.3% (14.3%) | yes |
| NQ | test | 36 | 7.0% (7.0%) | yes |
| MuSiQue | dev | 24 | 4.7% (4.7%) | no |
| Bamboogle | test | 1 | 0.2% (0.2%) | no |
| Total | 512 |
Naming: we will rename the benchmark to "Search-R1 QA" to avoid confusion with the existing SearchQA dataset.
5. [QA2 + CW1] Seed variance of the strongest baselines
QA2: "Can you run the strongest bank baseline with several seeds?"
CW1: "baselines have no repeated runs."
Across three seeds, TITRATE's reported success remains above the strongest baseline's seed mean on ALFWorld, WebShop and SearchQA. On ALFWorld, TITRATE is also 6–14× more stable across seeds than the baseline.
- Experiments: Due to the compute budget constraints, we reran the strongest bank baseline for each Qwen3-1.7B benchmark with 2 additional seeds in 300 training steps.
Table R4. BANK-FAIL across three seeds on ALFWorld / Qwen3-1.7B.
| Seed | Success Rate ↑ | Refresh | Analysis Call ↓ |
|---|---|---|---|
| 0 (paper Table 1) | 0.7617 ± 0.0509 | 865 | 3,559 |
| 1 | 0.6977 ± 0.0646 | 856 | 3,539 |
| 2 | 0.7953 ± 0.0521 | 835 | 3,540 |
| across-seed mean ± sd | 0.7516 ± 0.0496 | 852 ± 15 | 3,546 ± 11 |
| across-seed cv | 6.60% | 1.81% | 0.32% |
Table R5. BANK-FAIL across three seeds on WebShop / Qwen3-1.7B.
| Seed | Success Rate ↑ | Refresh | Analysis Call ↓ |
|---|---|---|---|
| 0 (paper Table 1) | 0.6256 ± 0.0432 | 269 | 4,481 |
| 1 | 0.6086 ± 0.0465 | 251 | 4,456 |
| 2 | 0.6641 ± 0.0361 | 298 | 4,487 |
| across-seed mean ± sd | 0.6328 ± 0.0284 | 273 ± 24 | 4,475 ± 16 |
| across-seed cv | 4.49% | 8.70% | 0.37% |
Table R6. BANK-STEP across three seeds on SearchQA / Qwen3-1.7B.
| Seed | Success Rate ↑ | Refresh | Analysis Call ↓ |
|---|---|---|---|
| 0 (paper Table 1) | 0.4199 ± 0.0196 | 4,285 | 4,301 |
| 1 | 0.3992 ± 0.0193 | 4,297 | 4,313 |
| 2 | 0.4279 ± 0.0121 | 4,277 | 4,312 |
| across-seed mean ± sd | 0.4157 ± 0.0148 | 4,286 ± 10 | 4,309 ± 7 |
| across-seed cv | 3.56% | 0.23% | 0.15% |
2) Observation 1: margins (Tables R4–R6). TITRATE exceeds the baseline’s three-seed mean by +3.51 pp on ALFWorld, +1.49 pp on WebShop and +1.35 pp on SearchQA.
Implication: these gains are comparable to seed variation, so we will report three-seed means in the revision.
3) Observation 2: stability on ALFWorld (Table R4; Table 18, Section G.1). BANK-FAIL’s across-seed cv in success is 6.60%, versus 0.46% / 1.06% for TITRATE in the higher / lower λ ranges, while TITRATE’s refresh count varies more (cv 5.7% / 7.4% vs. 1.8%).
Implication: TITRATE adapts its refreshes to each seed, yet reaches nearly the same final success.
6. [CW3] Choosing the λ range
CW3: "the λ range must be tuned per benchmark (the higher range makes WebShop collapse to zero)."
We agree that setting automatically is a valuable direction, but it requires careful design. For example, instead of a fixed range, could start from a single value and be adjusted by TITRATE’s own training signals (progress and RL–OPD conflict ), with a stop signal that halts increases before distillation becomes excessive. We will add this discussion to “Further Study for and Schedule” and leave the design to future work.
Response to Reviewer Akc7 (1/3)
We appreciate the reviewer Akc7’s careful reading and the constructive questions. We address each point in turn: comparison with published and reproduced methods and an RL-only baseline (QA1, QA5, CW2), analyzer cost on SearchQA (CW2), the Table 1 vs. Appendix G.2 difference (QA3, CW1), the SearchQA validation split and naming (QA4, suggestion), seed variance of the strongest baselines (QA2, CW1), and the λ range (CW3, suggestion).
1. [QA1 + QA5 + CW2] Main result: comparison with the existing methods
Conclusion: Among the methods without SFT in Table R1, TITRATE beats GRPO in all 9 settings and Skill-SD in 8 of 9 at matched steps, ranking first in three settings. It does so while building its skill bank from scratch, whereas Skill-SD, SDAR and the other Fixed baselines start from a pre-built o3 skill bank. To isolate skill handling from SFT, we build SEED without SFT (PER-TRAJ): TITRATE outperforms it in all 9 settings with 88.8–99.4% fewer analyzer calls. Against fixed skill-bank rules (BANK-*), TITRATE has the highest success in 8 of 9 settings, and the cheaper rules save cost only by suppressing refreshes. Overall, without SFT or a pre-built skill bank, TITRATE performs strongly at a fraction of the analyzer cost, and none of the simpler refresh rules we test can replace its controller.
Baseline Criteria: We compare policies evaluated without skills, which measures how much the policy itself improves. Therefore, methods whose policy uses skills at inference time (Skill1, SkillRL) are excluded. Methods with offline, pre-built skills (Fixed in Table R1) are included: they use a fixed SkillBank generated by o3, whereas TITRATE starts empty and builds its skills online. OPID and SEED generate skills online, but write new ones for each trajectory.
Evidence: Since rerunning every published method for 300 steps exceeds our compute budget, we reproduce OPID and GiGPO, build the comparison in five steps, each addressing one confound. We emphasize that evaluating at step 150 or 200 of TITRATE’s 300 training steps understates it: its cosine-annealed distillation weight is still mid-schedule, and its success rate is still rising.
Steps 1)–2) compare at matched training steps; step 3) compares our full schedule with SFT-initialized methods; steps 4)–5) are controlled comparisons with our manuscript’s baselines. For completeness, we also include TITRATE with the higher λ range [0.01, 0.1] (Section 4.3.1), omitting WebShop, where this range is unstable (Appendix G.2).
1) Matched steps vs published and reproduced methods (step 150, Table R1). TITRATE is above GRPO and Skill-GRPO in all 9 settings, Skill-SD in 8 of 9, RLSD and GRPO+OPSD in 7 of 9, and our reproduced OPID in 5 of 9. It is competitive with SDAR despite SDAR’s pre-built skills, and ahead of it on 3B and 7B SearchQA (43.8 vs. 43.4; 49.4 vs. 49.0).
Table R1. Success rate (%) at 150 training steps
Skill source: Fixed = a read-only skill library from SkillRL (generated by o3), not updated during training; Online = skills written by an analyzer (GPT-4.1) from the method’s own rollouts during training; selectively refreshed = a skill is rewritten only when TITRATE’s controller detects it is stale.