acceptodds
Under review as a conference paper at ICLR 2027

HARP: Unifying Reasoning Model and Long-Horizon Agents from Old Teacher Trajectories

Abstract

Reinforcement learning produces experts that solve problems and experts that carry out long-horizon tasks, while deployment calls for one policy that does both. Such consolidation is rarely done once, since a model family spans several sizes and products differ in budget and version. Routes that learn from live teachers, such as on-policy distillation, run the experts again whenever a new student is trained. Yet the experts' own RL training has already logged complete trajectories with their outcomes and behavior probabilities. Such a history records not only what an expert can do, but also how it came to do it. We present HARP, a fully offline framework that consolidates reasoning and long-horizon agency from these histories without running the experts again. The historical path curriculum releases experience along the teachers' own training stages, and the success-ratio budget turns successful trajectories into a provable lower bound on task success. A student then holds a non-decreasing floor on this bound for all acquired tasks. Once frozen, the history supports new students without expert inference or student rollouts for optimization. Experiments with Qwen3.5 at three scales show that HARP is (1) effective, recovering 103.6–105.5% of single-task RL gains on same-size pairs and lifting the 9B student of the 27B history from 22.5% to 39.3% on Terminal-Bench 2.1, (2) reusable, training 4B, 9B, and 27B students from one 27B history with zero new expert inference and 52.3% fewer marginal GPU hours than multi-teacher on-policy distillation at matched quality, and (3) self-sufficient, recovering 97.3% of the weakest-domain gain on eight GPUs with the experts offline. Our code and the record schema of teacher histories are available here.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Policy Distillation, Offline Learning, Multi-Task Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_gwPEOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

HARP reuses trajectories, outcomes, and behavior log-probabilities collected throughout teacher RL training to train students entirely offline. It combines a curriculum that progressively releases teacher-history data with a trajectory-ratio-based lower bound on success. Constraints and a joint checkpoint acceptance mechanism use this bound to protect previously acquired capabilities. Experiments cover mathematics, coding, terminal interaction, and web tasks, with analyses of cross-size transfer, repeated student production, acquisition order, retention, and computational cost.

Critical strengths
  1. The method addresses a concrete repeated-training workload. Reusing one frozen history across multiple students is practically meaningful. The cost analysis accounts for preparation, student training, and remaining teacher inference rather than equating zero teacher requests with overall efficiency.
  2. The component analyses are unusually informative. The ordering experiments control trajectory content and exposure, while the retention experiments compare protective objectives at similar acquisition gains. These designs provide stronger evidence than component-removal ablations alone.
Critical weaknesses
  1. Finite-sample constraint reliability remains an important limitation. Repeated optimization against reference panels can produce discrepancies between the empirical budget and actual success. The appendix includes a weak-support example where the estimated budget exceeds measured success. Independent audits help, but the frequency of such failures and sensitivity to panel size remain insufficiently characterized.
  2. The reuse benefits depend on demanding logging and compatibility conditions. The method requires complete interactions, accurate behavior probabilities, and provenance information. The demonstrated students also belong to a compatible model family. The evidence therefore supports reuse of suitably constructed historical assets more strongly than reuse of arbitrary existing logs or substantially different student architectures.
Questions for authors
  1. How do empirical constraint satisfaction and independently measured retention change as the reference-panel size decreases? How frequently does an accepted checkpoint exhibit a meaningful capability decline?
  2. Could you more prominently distinguish recovering a single-task HARP student's gains from recovering teacher capability, especially in cross-size compression settings?
  3. When existing logs lack behavior probabilities and additional preparation is necessary, how does including that cost change the break-even point for repeated student production?
5 comments
AuthorsOpenReview ↗

Response to Reviewer gwPE (1/3)

Thank you very much for your careful review and insightful questions, and for the time you have devoted to our paper. We truly appreciate your feedback and are glad to address each point below.

W1 / Q1. "Finite-sample constraint reliability remains an important limitation. Repeated optimization against reference panels can produce discrepancies between the empirical budget and actual success. … How do empirical constraint satisfaction and independently measured retention change as the reference-panel size decreases? How frequently does an accepted checkpoint exhibit a meaningful capability decline?"

Thank you for this important question. On independent panels, none of the 84 accepted (checkpoint, group) pairs of HARP on 9B→4B that follow a floor-setting commit shows a meaningful capability decline, and we are glad to report this measurement and how constraint satisfaction and retention change with the panel size.

Proposition 3 concerns population quantities, so we measure directly how often the implemented procedure lets acquired success decline. We use the budget on the reference panel as an optimization tool rather than as a certificate (Appendices B and H). Each group keeps 32 reference queries with 2 attempts per teacher run, 128 records per stage over two runs, recorded during teacher training and reused throughout student training. The discrepancies you describe arise from the estimation error of a finite panel and from the selection that repeated acceptance on the same panel introduces, and both can be measured without any expert, since the frozen history keeps the successes and failures of every stage (§2.1), not only the panel records.

Meaningful capability decline. To measure how frequently an accepted checkpoint declines, we take the accepted checkpoints of HARP on 9B→4B (three common seeds) and draw for each group and stage an independent panel of the same size from the logged trajectories of that stage, on queries outside the reference panel, so that the budget never uses it during training. We execute the student on the queries of both panels with four attempts each, and we call a decline meaningful when measured success falls below e−εe^{-\varepsilon} times its value at the commit that set the floor, with the tolerance ε=0.10\varepsilon=0.10 of Proposition 3 (Table 3). This never happens in the 84 accepted (checkpoint, group) pairs that follow a floor-setting commit. With floors set on the independent panel by the same rule (Eq. 8) at the same commits, none of these pairs falls below its floor either. The mean budget on the reused panel exceeds that on the independent one by only 0.46 points (52.59% against 52.13%), consistent with a small selection effect. These findings are consistent with Appendix G.1, where resetting the floors lowers the kept weakest prior gain from 95.3% to 76.2%.

Smaller panels weaken protection gradually, as weak support becomes more common. As suggested, we vary the panel size in the continued-training protocol of Appendix F.2 (9B→4B, reasoning pressure, three seeds), keeping nested subsets of 16 and 8 of the 32 reference queries with the same attempts and runs. The independent panels keep 128 records:

Records per stageWeakest prior gain kept at update 48New gain (points)Accepted pairs below independent floorWeakly supported panels
128 (paper)94.0%3.00.0%0.0%
6492.3%3.02.3%12.5%
3290.3%3.16.3%25.0%

At new gains of 3.0–3.1 points, independently measured retention declines from 94.0% to 92.3% and 90.3%, and the share of accepted pairs below the independent floor rises from 0.0% to 2.3% and 6.3%. At the same panel sizes, the share of weakly supported panels (fewer than 16 successes or fewer than 8 distinct successful queries, Appendix B) rises from 0.0% to 12.5% and 25.0%. Failures thus become more common together with weak support, the regime of the example in Appendix F.3, where a weakly supported reference yields a budget of 16.0%, above the measured success of 12.1%. Even 32 records per stage keep more of the weakest prior gain than the KL anchor, which keeps 82.6% at comparable new gains (§4.4).

AuthorsOpenReview ↗

Response to Reviewer gwPE (2/3)

Q2. "Could you more prominently distinguish recovering a single-task HARP student's gains from recovering teacher capability, especially in cross-size compression settings?"

Thank you for this valuable suggestion. We are glad to separate the two notions and to report recovery measured against the teachers.

Gain Recovery measures how much of a single-task HARP student's gain a joint student keeps, not how much teacher capability it recovers. The single-task reference is a HARP student trained on one domain from the same source (§4.1), so under compression it already bears the cost of absorbing a larger teacher. The gap decomposition of Appendix E.3 separates the two effects. For web tasks under the 27B history, the absorption gap between teacher and single-task student is 31.4, 15.6, and 6.5 points for the 4B, 9B, and 27B students, the integration gap from single-task to joint training is 0.5, −1.3, and −1.4 points, and the teacher-to-joint gaps total 31.9, 14.3, and 5.1 points. A Mean G above 100% thus means that a joint student exceeds its single-task references, not its teachers.

Measured against the teachers, recovery drops under compression for every method, including the single-task reference. We also report the teacher-referenced recovery GdT=(Jd(π)−Jd(π0))/(Jd(T)−Jd(π0))G_d^{T}=(J_d(\pi)-J_d(\pi_0))/(J_d(T)-J_d(\pi_0)), where Jd(T)J_d(T) is the mean test success of the two teachers of domain dd, with Mean and Min over the four domains as in §4.1 (%):

Teacher-referenced (Mean / Min)27B→27B27B→9B27B→4B
Single-task (HARP)62.0 / 41.750.6 / 34.430.4 / 21.7
MOPD66.5 / 49.551.0 / 33.629.1 / 20.0
D³-MOPD67.9 / 48.153.8 / 36.130.0 / 20.7
HARP64.9 / 46.852.5 / 36.929.9 / 21.2

HARP stays within 2.9 points of the single-task reference in Mean GTG^T at every size, while the Mean GTG^T of both falls from 64.9% and 62.0% at 27B to 29.9% and 30.4% at 4B. This indicates that what a compact student lacks relative to its teachers is lost mainly in absorbing a distant teacher rather than in consolidation. The recovery of 103.6–105.5% on same-size pairs therefore refers to the gains of single-task HARP students rather than to teacher capability.

AuthorsOpenReview ↗

Response to Reviewer gwPE (3/3)

W2 / Q3. "The reuse benefits depend on demanding logging and compatibility conditions. The method requires complete interactions, accurate behavior probabilities, and provenance information. The demonstrated students also belong to a compatible model family. … When existing logs lack behavior probabilities and additional preparation is necessary, how does including that cost change the break-even point for repeated student production?"

Thank you for raising this practical point. We are glad to clarify which fields are required and to report how backfilling the behavior probabilities changes the break-even point.

The required fields are those that the experts' RL training already logs. This training records complete trajectories with their outcomes and behavior probabilities (abstract, §2.1), each record carries the run and step that produced it (Eq. 2), and the behavior log-probabilities are recorded at generation time under the actual sampling configuration (§2.1). We release the record schema that lists these fields. If a history lacks the probabilities, we count the teacher computation that fills them as preparation and report the result as reuse after backfill (Appendix H).

Backfill is a one-time preparation cost, so it moves the break-even point rather than the saving per student. Backfill scores each logged trajectory once with the snapshot that generated it, without any generation, and scoring the complete 27B history in this way takes 81.9 GPU hours. In the ledger of Proposition 4, it adds to the preparation AHARPA_{\mathrm{HARP}}, so HARP with backfill costs less than a comparator after KK students exactly when the backfill cost is below the comparator's cumulative headroom, that is, its preparation and marginal costs minus those of HARP (Table 6, Appendix D):

ComparatorHeadroom after 1 / 2 / 3 students (GPU h)Students to break even with backfill
MOPD144.9 / 296.6 / 580.71
ReOPD35.8 / 71.5 / 86.53
D³-MOPD53.8 / 121.9 / 101.22

Without backfill, HARP costs less from the first student against all three comparators. With backfill, HARP still costs less than MOPD from the first student and overtakes D³-MOPD at the second and ReOPD at the third, while the expert computation that it avoids keeps growing with every new student (Proposition 4).

On compatibility, only the trajectory log-ratio relies on the shared tokenizer and chat template. The recorded behavior log-probabilities enter only through the log-ratio ZZ of Eq. 3, which compares them with the student's log-likelihood of the same action tokens and drives the budget (Eq. 6) and the truncated ratio of negative advantages (Eq. 5). The release coordinate, the advantages, the positive branch of Eq. 5, and the panel weights read only text, rewards, outcomes, and provenance. A student with another tokenizer or template can re-render the stored interactions in its own format, but the log-ratio then needs behavior probabilities on its own action tokens, which the recorded ones provide directly only under a shared tokenizer. Our experiments cover students that share the tokenizer and template, and Appendix H scopes our claims accordingly.

Thank you again for your thoughtful questions, which helped us characterize the reliability and the cost of reuse more precisely. We hope these responses address your concerns, and we would be very happy to discuss any further questions.

Reviewer_RamZOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors proposed HARP, training students offline from trajectories recorded during specialist teachers' reinforcement learning. It combines a curriculum based on teacher training progress with success-ratio constraints intended to preserve acquired capabilities. Experiments evaluate capability consolidation and reuse across student sizes on math, code, terminal, and web tasks.

Critical strengths
  • Reusing specialist training histories addresses a practical consolidation problem without further teacher inference.
  • The evaluation method, model coverage is broad.
Critical weaknesses
  1. The supplied implementation does not maintain the invariant used in Proposition 3. SuccessBudget.commit raises the floor only when the reference advances; improvements under an unchanged or final reference leave it unchanged. The proof instead assumes a floor reflecting the acquired budget (Appendix C.3). This weakens the claimed retention guarantee, although the population lower-bound formula itself remains valid.

  2. Novelty appears incremental. HARP's contribution over REGEN lies mainly in combining a history curriculum with success-budget retention. A learner-adaptive curriculum comparison using the same logs is missing, leaving the curriculum's advantage over prior adaptive approaches unclear.

  3. The GPU-hour comparison does not isolate HARP-specific efficiency. Avoiding new rollouts and teacher inference is already shared by REGEN (Appendix H), so the 52.3% marginal saving over MOPD does not by itself establish a new efficiency advantage. At fixed updates, HARP uses 73.3–92.3% more GPU hours than REGEN (Appendix G). Although REGEN misses the chosen quality target, comparisons with offline baselines at matched compute budgets are needed to assess whether HARP's quality gains justify this overhead.

Questions for authors
  1. Did the reported experiments use the supplied floor-update code? Please reconcile it with Proposition 3 and report corrected retention results if necessary.
  2. Can you provide the promised analysis scripts, run configurations, and per-seed results?
  3. How well do the reused-panel budgets track independently measured task success at accepted checkpoints?
  4. Do the recovery and GPU-hour conclusions still hold when qualification targets are defined independently of HARP, across several quality thresholds—including thresholds REGEN reaches—rather than relative to single-domain HARP gains?
7 comments
AuthorsOpenReview ↗

Response to Reviewer RamZ (1/6)

Thank you very much for your careful review and insightful questions, and for the time you have devoted to our paper. We truly appreciate your feedback and are glad to address each point below.

W1 / Q1. "The supplied implementation does not maintain the invariant used in Proposition 3. SuccessBudget.commit raises the floor only when the reference advances; improvements under an unchanged or final reference leave it unchanged. The proof instead assumes a floor reflecting the acquired budget (Appendix C.3). … Did the reported experiments use the supplied floor-update code? Please reconcile it with Proposition 3 and report corrected retention results if necessary."

Thank you for examining the implementation in such detail. Our experiments follow the floor rule that you describe, and we are glad to reconcile it with Proposition 3 and to report retention under a rule that raises the floor at every commit.

A floor changes only when its group becomes supported or advances its reference. This is the rule of §2.3. A floor is set at the current budget minus ε\varepsilon when a group becomes supported and raised by Eq. 8 when the group advances its reference. At an accepted commit that keeps the reference, whether intermediate or final, the floor stays unchanged.

Proposition 3 holds as stated for every commit that sets the floor. The proof in Appendix C.3 starts from fg≥log⁡Bg(πs)−εf_g\ge\log\mathcal B_g(\pi_s)-\varepsilon after commit ss, which holds at every commit that sets the floor, since Eq. 8 takes a maximum and a newly supported group starts at its budget minus ε\varepsilon. The remaining steps use only that floors never decrease, that an accepted candidate meets all current floors and advances a reference only if its budget does not drop, and that Proposition 1 holds for any reference. For a floor-setting commit ss and any later accepted student πt\pi_t,

Jg(πt) ≥ Bcg(t),g(πt) ≥ efg(t) ≥ efg(s) ≥ e−εBcg(s),g(πs).J_g(\pi_t)\ \ge\ \mathcal B_{c_g^{(t)},g}(\pi_t)\ \ge\ e^{f_g^{(t)}}\ \ge\ e^{f_g^{(s)}}\ \ge\ e^{-\varepsilon}\mathcal B_{c_g^{(s)},g}(\pi_s).

At a commit that keeps the reference, the guarantee therefore holds relative to the last commit that set the floor, so an improvement under an unchanged or final reference is protected only up to that earlier level, while the lower bound of Proposition 1 is unaffected.

Corrected retention results. To report retention under a rule for which Proposition 3 holds at all commits as originally stated, we apply Eq. 8 at every accepted commit and keep the current reference in it when the reference does not advance. Since our retention results (§4.4) come from the continued-training protocol of Appendix F.2, we rerun its budget arm with this rule under both pressures (9B→4B, three seeds):

Floor updateWeakest prior gain kept, reasoning pressureNew gain, reasoning pressure (points)Weakest prior gain kept, agent pressureNew gain, agent pressure (points)
At reference changes (reported)94.0%3.095.3%4.0
At every accepted commit93.6%3.195.7%4.1

At comparable new gains, the weakest prior gain kept under the every-commit rule differs from that of the reported rule by at most 0.4 points, lower under reasoning pressure and higher under agent pressure. This indicates that the reported retention does not hinge on the scope of the guarantee.

AuthorsOpenReview ↗

Response to Reviewer RamZ (2/6)

Q3. "How well do the reused-panel budgets track independently measured task success at accepted checkpoints?"

Thank you for this thoughtful question. We are glad to report how the reused-panel budgets compare with independently measured success and with budgets on an independent panel.

Reused-panel budgets stay below measured success on average and exceed independent-panel budgets by only 0.46 points. We use the budget on the reference panel as an optimization tool rather than as a certificate (Appendices B and H). Each group keeps 32 reference queries with 2 attempts per teacher run, 128 records per stage over two runs, recorded during teacher training and reused throughout student training, so acceptance consults the same panel repeatedly. To measure the resulting discrepancy without any expert, we draw for each group and stage an independent panel of the same size from the logged trajectories of that stage, on queries outside the reference panel, so that the budget never uses it during training. For the accepted checkpoints of HARP on 9B→4B (three common seeds), we compute the budgets on both panels, execute the student on the queries of both panels with four attempts each, and on the independent panel set floors by the same rule (Eq. 8) at the same commits. Means and rank correlations are over all 180 accepted (checkpoint, group) pairs, and the floor check covers the 84 pairs that follow the commit setting their floor:

PanelMean budget − measured success (points)Rank correlation of budget and measured successAccepted pairs below floor
Reused reference panel−2.550.980 (by acceptance)
Independent panel−2.700.990 of 84

The mean budget is 52.59% on the reused panel and 52.13% on the independent one, a difference of 0.46 points that is consistent with a small selection effect of repeated acceptance. On both panels, the budgets follow measured success closely in rank across pairs. In none of the 84 pairs that follow a floor-setting commit does measured success drop below e−εe^{-\varepsilon} times its value at that commit (ε=0.10\varepsilon=0.10, Table 3), and none of them falls below its floor on the independent panel. These findings are consistent with the calibration of Appendix F.3, where compatible references yield budgets of 20.0–69.9% against measured success of 24.6–74.9%, and with Appendix G.1, where resetting the floors lowers the kept weakest prior gain from 95.3% to 76.2%.

AuthorsOpenReview ↗

Response to Reviewer RamZ (3/6)

W2. "Novelty appears incremental. HARP's contribution over REGEN lies mainly in combining a history curriculum with success-budget retention. A learner-adaptive curriculum comparison using the same logs is missing, leaving the curriculum's advantage over prior adaptive approaches unclear."

Thank you for this valuable suggestion. We are glad to clarify what HARP adds over REGEN and to report a comparison with learner-adaptive curricula on the same logs.

HARP changes how an offline learner uses a history, beyond the losses it combines. REGEN replays the pooled history uniformly on single-turn tasks and leaves stage information to future work (Appendix A). We read the optimization order of the history as a release schedule calibrated by context, feedback depth, and generation length, whose stride Proposition 2 grounds in the local tightness of the bound (§2.2, §3). We further turn successful trajectories and their behavior probabilities into a lower bound on task success that the log alone can compute (Proposition 1), held by jointly committed floors for multi-turn agents (§2.3). Among the log-only methods of Table 2, HARP is the only one that follows the teachers' order, protects what has been acquired, and is validated on multi-turn agents.

Each part is needed. On 9B→4B, the curriculum alone and the budget alone lift the Mean G of shuffled replay from 71.0% to 85.6% and 89.2%, the full method reaches 104.4% (§4.4), and a token-mean budget, which gives up the trajectory-level bound, drops to 88.3% (Appendix E.2). With content and exposure controlled, the historical order reaches the learning milestone with 73.4% of the compute of matched shuffling (Figure 4), and under reasoning pressure the budget keeps 94.0% of the weakest prior gain against 82.6% for a KL anchor at comparable new gains.

Learner-adaptive curricula on the same logs. To test whether a curriculum driven by the student's current state uses the same logs better, we compare the historical path with two learner-adaptive curricula from prior work, both reading only the frozen history and the student. Learning-progress release, after Graves et al. (2017), opens next the parts of the history on which the student's negative log-likelihood has recently decreased most. Policy-similarity release, after COIL, opens next the parts whose records are closest to the current student by the per-token log-ratio ∣Zi∣/Ni\lvert Z_i\rvert/N_i. Both open records at the same pace as our clock, so the rules differ only in which records open. We first compare the three release rules in the order protocol of Appendix F.1, where the budget is off and the trajectory multiset and exposure are matched:

Release on 9B→4BMilestone cost (% of matched shuffle)
Learning progress88.8
Policy similarity82.6
Historical path (HARP)73.4

We then train policy similarity, the adaptive rule with the lower milestone cost, end to end with the budget on the three common seeds. Its reference panels follow the historical stages as in HARP, so its budget uses the same references. It reaches 100.7% Mean G and 97.7% Min G, against 104.4% and 100.5% for HARP on the same seeds, and scoring the history at each refresh adds 10.1% to its training compute. The historical path thus reaches the milestone with less compute than both adaptive rules and ends higher than the one trained end to end, which suggests that the teachers' own order carries information that these state-driven rules do not exploit.

Reviewer_zqzFOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

HARP trains a generalist student from the frozen reinforcement-learning histories of domain-specific teachers, without querying those teachers or collecting student rollouts for training. It releases logged trajectories according to the teachers’ training stages and uses a success-ratio budget to retain acquired capabilities. The paper proves a population lower bound on task success, then evaluates the method across six teacher–student size pairs and five reasoning and agent benchmarks. Ablations examine acquisition order and retention; a resource study measures the cost of training three students from one history.

Critical strengths
  1. Strong evidence for both components. The component ablations, matched-content ordering intervention, and continued-training comparisons help attribute the gains to the curriculum and retention budget. This substantially strengthens the empirical contribution.

  2. A practically meaningful reuse setting. One frozen 27B history trains 4B, 9B, and 27B students without further expert inference. The paper reports quality-matched marginal GPU costs and states the access and accounting assumptions, making the reuse claim consequential and assessable.

Critical weaknesses
  1. The empirical floors do not inherit the population guarantee. Proposition 1 concerns population quantities, while checkpoint acceptance uses estimates from small, repeatedly consulted panels. The paper acknowledges a weakly supported case where the estimated budget exceeds independently measured success (16.0% versus 12.1%). This matters because reliable retention is a central claim; the theoretical result alone does not establish it for the implemented procedure.

  2. Transfer beyond the tested model family remains uncertain. All cross-size experiments use Qwen3.5 models with the same tokenizer and chat template, and the three-student reuse study draws on one 27B history. This limits the evidence for how broadly a frozen history can serve later or substantially different students, although it does not undermine the reported setting.

Questions for authors
  1. Across training, how often would a committed checkpoint fail its floor when the budget is recomputed on an independent held-out reference panel? How does that frequency change with panel size or a confidence-adjusted floor? This would materially clarify the practical force of the retention guarantee.

  2. Which parts of the method require the shared tokenizer and chat template, and what evidence supports reuse when a later student differs in either respect? A concrete account of the required adaptation would clarify the scope of the “one history, many students” claim.

2 comments
AuthorsOpenReview ↗

Response to Reviewer zqzF (1/2)

Thank you very much for your careful review and insightful questions, and for the time you have devoted to our paper. We truly appreciate your feedback and are glad to address each point below.

W1 / Q1. "The empirical floors do not inherit the population guarantee. Proposition 1 concerns population quantities, while checkpoint acceptance uses estimates from small, repeatedly consulted panels. … Across training, how often would a committed checkpoint fail its floor when the budget is recomputed on an independent held-out reference panel? How does that frequency change with panel size or a confidence-adjusted floor?"

Thank you for raising this important point. On an independent held-out panel, none of the 84 accepted (checkpoint, group) pairs of HARP on 9B→4B that follow a floor-setting commit fails its floor, and we are glad to report this measurement and how it changes with the panel size.

Since Propositions 1 and 3 concern population quantities, the practical question is how often the implemented procedure lets acquired success decline. We use the budget on the reference panel as an optimization tool rather than as a certificate (Appendices B and H). Each group keeps 32 reference queries with 2 attempts per teacher run, that is, 128 records per stage over two runs, recorded during teacher training and reused throughout student training. Two effects separate the implemented floors from the population guarantee, namely the estimation error of a finite panel and the selection that arises because every acceptance consults the same panel. Both can be measured without any expert, since the frozen history keeps the successes and failures of every stage (§2.1), not only the panel records.

Independent held-out panel. To measure how often a committed checkpoint fails its floor, we take the accepted checkpoints of HARP on 9B→4B (three common seeds) and draw for each group and stage an independent panel of the same size from the logged trajectories of that stage, on queries outside the reference panel, so that the budget never uses it during training. On it we recompute the budgets and set the floors by the same rule (Eq. 8) at the same commits, which lets the check compare one panel with itself over time. We also execute the student on the queries of both panels with four attempts each. Means are over all 180 accepted (checkpoint, group) pairs, and the floor check covers the 84 pairs that follow the commit setting their floor:

PanelMean budget − measured success (points)Accepted pairs below floor
Reused reference panel−2.550 (by acceptance)
Independent panel−2.700 of 84

None of the 84 pairs falls below its floor on the independent panel, and in none of them does measured success drop below e−εe^{-\varepsilon} times its value at the commit that set the floor (ε=0.10\varepsilon=0.10, Table 3). Budgets on both panels stay below measured success on average, and the mean budget on the reused panel exceeds that on the independent one by only 0.46 points (52.59% against 52.13%), consistent with a small selection effect. These findings are consistent with Appendix G.1, where resetting the floors lowers the kept weakest prior gain from 95.3% to 76.2%.

Protection weakens gradually as the panel shrinks. As suggested, we vary the panel size in the continued-training protocol of Appendix F.2 (9B→4B, reasoning pressure, three seeds), keeping nested subsets of 16 and 8 of the 32 reference queries with the same attempts and runs. The independent panels keep 128 records:

Records per stageWeakest prior gain kept at update 48New gain (points)Accepted pairs below independent floorWeakly supported panels
128 (paper)94.0%3.00.0%0.0%
6492.3%3.02.3%12.5%
3290.3%3.16.3%25.0%

At comparable new gains, retention declines by 1.7 and 3.7 points as failures and weakly supported panels become more common, the regime of the example in Appendix F.3, where a weakly supported reference yields a budget of 16.0%, above the measured success of 12.1%. Even 32 records per stage keep more of the weakest prior gain than the KL anchor, which keeps 82.6% at comparable new gains (§4.4).

AuthorsOpenReview ↗

Response to Reviewer zqzF (2/2)

W2 / Q2. "Transfer beyond the tested model family remains uncertain. All cross-size experiments use Qwen3.5 models with the same tokenizer and chat template, and the three-student reuse study draws on one 27B history. … Which parts of the method require the shared tokenizer and chat template, and what evidence supports reuse when a later student differs in either respect? A concrete account of the required adaptation would clarify the scope of the “one history, many students” claim."

Thank you for this detailed question. We are glad to give a concrete account of which parts rely on the shared tokenizer and chat template and of what a different student would need.

Only the trajectory log-ratio relies on the shared tokenizer and chat template. The recorded behavior log-probability ℓμ\ell^{\mu} stays valid for 4B, 9B, and 27B students because the three sizes share one tokenizer and chat template, so replay scores the same action tokens at any size (§2.4, Appendix D). It enters only through the log-ratio Z=ℓθ(τ)−ℓμZ=\ell_\theta(\tau)-\ell^{\mu} of Eq. 3, which drives the budget (Eq. 6) and the truncated ratio that tempers negative advantages in the base learner (Eq. 5). All other parts read only the logged text, rewards, outcomes, and provenance. The release coordinate of Eq. 4 uses teacher steps and within-domain ranks of the context length and feedback depth of a fixed reference policy. The advantages, the positive branch of Eq. 5, the panel weights, and the success indicators need no teacher probability.

A later student with another tokenizer or template needs its own rendering of the history and behavior probabilities on its own action tokens. Since the frozen history stores complete actions and observations (§2.4), the logged interactions can be re-rendered in the student's chat template and tokens, and the length scales of Eq. 4, counted in tokens (Table 3), can be re-measured, while teacher steps, groups, and ranks carry over unchanged. The log-ratio, however, compares the student with the generating snapshot on the same action tokens, which the recorded probabilities provide directly only under a shared tokenizer, so this part needs an additional conversion that our experiments do not cover.

Our evidence covers students that share the tokenizer and template. Within this family, each of the three histories trains a student of its own size, and the 9B and 27B histories also train smaller students (9B→4B, 27B→9B, 27B→4B). The 4B student of the 9B history exceeds the initial 9B model on both agentic benchmarks, with 30.5% against 22.5% on Terminal-Bench 2.1 and 42.5% against 30.9% on WA-TG20 (§4.2). Students with another tokenizer lie outside our experiments, and Appendix H scopes our claims to compatible students.

Thank you again for your thoughtful questions, which helped us examine the empirical floors and the scope of reuse more closely. We hope these responses address your concerns, and we would be very happy to discuss any further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to the Area Chair and All Reviewers

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair for handling our submission, and the reviewers (zqzF, RamZ, gwPE) for their time, careful reading, and insightful, constructive feedback. We have answered each reviewer with the complete new results and incorporated the key ones into the revised paper, where all changes are highlighted in blue.


Strengths recognized by the reviewers

We are deeply encouraged that the reviewers recognize:

  • A practical reuse setting. Reusing specialist training histories "addresses a practical consolidation problem without further teacher inference" (RamZ), and "reusing one frozen history across multiple students is practically meaningful" (gwPE).
  • Informative evidence and accounting. The reviewers find "strong evidence for both components" (zqzF) and note that "the component analyses are unusually informative" (gwPE), that "the evaluation method, model coverage is broad" (RamZ), and that the accounting makes "the reuse claim consequential and assessable" (zqzF).

Main concerns and how we have addressed them

1. Reliability of the empirical floors · zqzF RamZ gwPE

Concern. Whether floors estimated on small, reused panels protect acquired success, and how panel size matters.

  • Independent panels. On 9B→4B, none of the 84 accepted (checkpoint, group) pairs after a floor-setting commit falls below its floor or declines meaningfully on panels that the budget never uses.
  • Panel size. With 64 and 32 instead of 128 records per stage, retention declines gradually from 94.0% to 92.3% and 90.3%, above the 82.6% of the KL anchor.

In the revised paper: §3 and Appendix H; new Appendix I (independent panels, panel size).

2. Floor update and Proposition 3 · RamZ

Concern. The implemented floor rises only when the reference advances, unlike the invariant in the proof of Proposition 3.

  • Reconciled guarantee. Proposition 3 holds as stated at every floor-setting commit under the rule our experiments follow (§2.3).
  • Corrected retention. Raising the floor at every accepted commit keeps 93.6% and 95.7% of the weakest prior gain under the two pressures, against 94.0% and 95.3% for the reported rule.

In the revised paper: §2.3, Algorithm 1, §3, and Appendix C.3 (floor-setting commits); new Appendix I (every-commit floors).

3. Novelty and learner-adaptive curricula · RamZ

Concern. What HARP adds over REGEN, and how its curriculum compares with learner-adaptive ones on the same logs.

  • What HARP adds. Release along the teachers' optimization order and a log-computable success bound held by joint floors, both needed in the ablations.
  • Adaptive release. Learning-progress and policy-similarity release reach the milestone with 88.8% and 82.6% of the compute of matched shuffling, against 73.4% for the historical path. End to end, policy similarity reaches 100.7% Mean G, against 104.4% for HARP on the same seeds.

In the revised paper: §4.4 and Appendix A; new Appendix K (learner-adaptive release).

4. Compute-matched and HARP-independent comparisons · RamZ gwPE

Concern. Whether HARP's quality justifies its GPU-hour overhead, and whether its conclusions hold for HARP-independent targets.

  • Matched GPU hours. With the GPU hours of HARP on 9B→4B, REGEN and REGEN-KL reach 77.6% and 88.9% Mean G, against 104.6% for HARP.
  • HARP-independent targets. At the final levels of REGEN, REGEN-KL, and MOPD, HARP qualifies the three 27B-history students with 114.3, 214.3, and 378.5 marginal GPU hours, fewer than any other method compared.
  • Teacher-referenced recovery. HARP leads all offline baselines under any increasing, shared normalization without HARP, and on the 27B history stays within 2.9 points of the single-task reference in Mean GTG^T.

In the revised paper: §4.1 and Table 1 (reference of Gain Recovery); §4.3; new Appendix J (matched compute, HARP-independent targets).

5. Logging and compatibility · zqzF gwPE

Concern. Which parts need the shared tokenizer and template, and how backfilling behavior probabilities shifts the break-even point.

  • Compatibility. Only the trajectory log-ratio uses the recorded behavior probabilities. The other parts read only text, rewards, outcomes, and provenance.
  • Backfill. Scoring the 27B history once takes 81.9 GPU hours, after which HARP still costs less than MOPD from the first student, D³-MOPD from the second, and ReOPD from the third.

In the revised paper: §2.4 and Appendix H; new Appendix J (backfill).


We are sincerely grateful to the Area Chair and the reviewers for their time and effort, which have helped strengthen the paper, and we would be very happy to discuss any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.