- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
HARP reuses trajectories, outcomes, and behavior log-probabilities collected throughout teacher RL training to train students entirely offline. It combines a curriculum that progressively releases teacher-history data with a trajectory-ratio-based lower bound on success. Constraints and a joint checkpoint acceptance mechanism use this bound to protect previously acquired capabilities. Experiments cover mathematics, coding, terminal interaction, and web tasks, with analyses of cross-size transfer, repeated student production, acquisition order, retention, and computational cost.
Critical strengths
- The method addresses a concrete repeated-training workload. Reusing one frozen history across multiple students is practically meaningful. The cost analysis accounts for preparation, student training, and remaining teacher inference rather than equating zero teacher requests with overall efficiency.
- The component analyses are unusually informative. The ordering experiments control trajectory content and exposure, while the retention experiments compare protective objectives at similar acquisition gains. These designs provide stronger evidence than component-removal ablations alone.
Critical weaknesses
- Finite-sample constraint reliability remains an important limitation. Repeated optimization against reference panels can produce discrepancies between the empirical budget and actual success. The appendix includes a weak-support example where the estimated budget exceeds measured success. Independent audits help, but the frequency of such failures and sensitivity to panel size remain insufficiently characterized.
- The reuse benefits depend on demanding logging and compatibility conditions. The method requires complete interactions, accurate behavior probabilities, and provenance information. The demonstrated students also belong to a compatible model family. The evidence therefore supports reuse of suitably constructed historical assets more strongly than reuse of arbitrary existing logs or substantially different student architectures.
Questions for authors
- How do empirical constraint satisfaction and independently measured retention change as the reference-panel size decreases? How frequently does an accepted checkpoint exhibit a meaningful capability decline?
- Could you more prominently distinguish recovering a single-task HARP student's gains from recovering teacher capability, especially in cross-size compression settings?
- When existing logs lack behavior probabilities and additional preparation is necessary, how does including that cost change the break-even point for repeated student production?
5 comments
Response to Reviewer gwPE (2/3)
Q2."Could you more prominently distinguish recovering a single-task HARP student's gains from recovering teacher capability, especially in cross-size compression settings?"
Thank you for this valuable suggestion. We are glad to separate the two notions and to report recovery measured against the teachers.
Gain Recovery measures how much of a single-task HARP student's gain a joint student keeps, not how much teacher capability it recovers. The single-task reference is a HARP student trained on one domain from the same source (§4.1), so under compression it already bears the cost of absorbing a larger teacher. The gap decomposition of Appendix E.3 separates the two effects. For web tasks under the 27B history, the absorption gap between teacher and single-task student is 31.4, 15.6, and 6.5 points for the 4B, 9B, and 27B students, the integration gap from single-task to joint training is 0.5, −1.3, and −1.4 points, and the teacher-to-joint gaps total 31.9, 14.3, and 5.1 points. A Mean G above 100% thus means that a joint student exceeds its single-task references, not its teachers.
Measured against the teachers, recovery drops under compression for every method, including the single-task reference. We also report the teacher-referenced recovery , where is the mean test success of the two teachers of domain , with Mean and Min over the four domains as in §4.1 (%):
| Teacher-referenced (Mean / Min) | 27B→27B | 27B→9B | 27B→4B |
|---|---|---|---|
| Single-task (HARP) | 62.0 / 41.7 | 50.6 / 34.4 | 30.4 / 21.7 |
| MOPD | 66.5 / 49.5 | 51.0 / 33.6 | 29.1 / 20.0 |
| D³-MOPD | 67.9 / 48.1 | 53.8 / 36.1 | 30.0 / 20.7 |
| HARP | 64.9 / 46.8 | 52.5 / 36.9 | 29.9 / 21.2 |
HARP stays within 2.9 points of the single-task reference in Mean at every size, while the Mean of both falls from 64.9% and 62.0% at 27B to 29.9% and 30.4% at 4B. This indicates that what a compact student lacks relative to its teachers is lost mainly in absorbing a distant teacher rather than in consolidation. The recovery of 103.6–105.5% on same-size pairs therefore refers to the gains of single-task HARP students rather than to teacher capability.
Response to Reviewer gwPE (3/3)
W2 / Q3."The reuse benefits depend on demanding logging and compatibility conditions. The method requires complete interactions, accurate behavior probabilities, and provenance information. The demonstrated students also belong to a compatible model family. … When existing logs lack behavior probabilities and additional preparation is necessary, how does including that cost change the break-even point for repeated student production?"
Thank you for raising this practical point. We are glad to clarify which fields are required and to report how backfilling the behavior probabilities changes the break-even point.
The required fields are those that the experts' RL training already logs. This training records complete trajectories with their outcomes and behavior probabilities (abstract, §2.1), each record carries the run and step that produced it (Eq. 2), and the behavior log-probabilities are recorded at generation time under the actual sampling configuration (§2.1). We release the record schema that lists these fields. If a history lacks the probabilities, we count the teacher computation that fills them as preparation and report the result as reuse after backfill (Appendix H).
Backfill is a one-time preparation cost, so it moves the break-even point rather than the saving per student. Backfill scores each logged trajectory once with the snapshot that generated it, without any generation, and scoring the complete 27B history in this way takes 81.9 GPU hours. In the ledger of Proposition 4, it adds to the preparation , so HARP with backfill costs less than a comparator after students exactly when the backfill cost is below the comparator's cumulative headroom, that is, its preparation and marginal costs minus those of HARP (Table 6, Appendix D):
| Comparator | Headroom after 1 / 2 / 3 students (GPU h) | Students to break even with backfill |
|---|---|---|
| MOPD | 144.9 / 296.6 / 580.7 | 1 |
| ReOPD | 35.8 / 71.5 / 86.5 | 3 |
| D³-MOPD | 53.8 / 121.9 / 101.2 | 2 |
Without backfill, HARP costs less from the first student against all three comparators. With backfill, HARP still costs less than MOPD from the first student and overtakes D³-MOPD at the second and ReOPD at the third, while the expert computation that it avoids keeps growing with every new student (Proposition 4).
On compatibility, only the trajectory log-ratio relies on the shared tokenizer and chat template. The recorded behavior log-probabilities enter only through the log-ratio of Eq. 3, which compares them with the student's log-likelihood of the same action tokens and drives the budget (Eq. 6) and the truncated ratio of negative advantages (Eq. 5). The release coordinate, the advantages, the positive branch of Eq. 5, and the panel weights read only text, rewards, outcomes, and provenance. A student with another tokenizer or template can re-render the stored interactions in its own format, but the log-ratio then needs behavior probabilities on its own action tokens, which the recorded ones provide directly only under a shared tokenizer. Our experiments cover students that share the tokenizer and template, and Appendix H scopes our claims accordingly.
Thank you again for your thoughtful questions, which helped us characterize the reliability and the cost of reuse more precisely. We hope these responses address your concerns, and we would be very happy to discuss any further questions.
Response to Reviewer gwPE (1/3)
Thank you very much for your careful review and insightful questions, and for the time you have devoted to our paper. We truly appreciate your feedback and are glad to address each point below.
Thank you for this important question. On independent panels, none of the 84 accepted (checkpoint, group) pairs of HARP on 9B→4B that follow a floor-setting commit shows a meaningful capability decline, and we are glad to report this measurement and how constraint satisfaction and retention change with the panel size.
Proposition 3 concerns population quantities, so we measure directly how often the implemented procedure lets acquired success decline. We use the budget on the reference panel as an optimization tool rather than as a certificate (Appendices B and H). Each group keeps 32 reference queries with 2 attempts per teacher run, 128 records per stage over two runs, recorded during teacher training and reused throughout student training. The discrepancies you describe arise from the estimation error of a finite panel and from the selection that repeated acceptance on the same panel introduces, and both can be measured without any expert, since the frozen history keeps the successes and failures of every stage (§2.1), not only the panel records.
Meaningful capability decline. To measure how frequently an accepted checkpoint declines, we take the accepted checkpoints of HARP on 9B→4B (three common seeds) and draw for each group and stage an independent panel of the same size from the logged trajectories of that stage, on queries outside the reference panel, so that the budget never uses it during training. We execute the student on the queries of both panels with four attempts each, and we call a decline meaningful when measured success falls below e−ε times its value at the commit that set the floor, with the tolerance ε=0.10 of Proposition 3 (Table 3). This never happens in the
84accepted (checkpoint, group) pairs that follow a floor-setting commit. With floors set on the independent panel by the same rule (Eq. 8) at the same commits, none of these pairs falls below its floor either. The mean budget on the reused panel exceeds that on the independent one by only0.46points (52.59% against 52.13%), consistent with a small selection effect. These findings are consistent with Appendix G.1, where resetting the floors lowers the kept weakest prior gain from 95.3% to 76.2%.Smaller panels weaken protection gradually, as weak support becomes more common. As suggested, we vary the panel size in the continued-training protocol of Appendix F.2 (9B→4B, reasoning pressure, three seeds), keeping nested subsets of 16 and 8 of the 32 reference queries with the same attempts and runs. The independent panels keep 128 records:
At new gains of 3.0–3.1 points, independently measured retention declines from 94.0% to
92.3%and90.3%, and the share of accepted pairs below the independent floor rises from 0.0% to2.3%and6.3%. At the same panel sizes, the share of weakly supported panels (fewer than 16 successes or fewer than 8 distinct successful queries, Appendix B) rises from 0.0% to12.5%and25.0%. Failures thus become more common together with weak support, the regime of the example in Appendix F.3, where a weakly supported reference yields a budget of 16.0%, above the measured success of 12.1%. Even 32 records per stage keep more of the weakest prior gain than the KL anchor, which keeps 82.6% at comparable new gains (§4.4).