- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Waypoint-OPD, which transfers a teacher's RL-driven policy changes via an intermediate checkpoint from the same training run. For each student-generated prefix, it measures how the teacher's preferences over the student's most likely next tokens shifted in the first versus second half of training. When the two shifts agree in direction, the full start-to-finish change is trusted; when they conflict, it is discounted or dropped. This agreement signal gates the initial-to-final policy shift, and the student is trained on dense top-k candidate scores rescaled within each batch, with a KL penalty anchoring it to its initialization.
Critical strengths
A focused and clearly specified methodological contribution. The intermediate checkpoint provides information unavailable from the endpoint pair while preserving the endpoint shift as the transferred signal. The method is straightforward to implement and requires only one additional frozen-teacher scoring pass, making it useful for reusing existing RL checkpoints.
Critical weaknesses
-
The gate helps, but why it helps remains unclear. The no-gate and shuffled-gate experiments provide useful evidence that the proposed weighting works. However, they do not show whether temporal agreement is better than simpler choices, such as weighting signals by their magnitude or the student’s confidence. They also do not separately test whether the benefit comes from removing conflicting signals or reducing the weight of partially aligned signals.
-
The reported gains need uncertainty estimates. The mathematical average improves over Direct-OPD by 1.59–2.18 percentage points, but the paper provides no confidence intervals or variation across repeated training runs. Generating 32 answers per problem does not address training variability or the limited number of distinct problems. AIME24 is also used to select checkpoints and included in the reported average. The learning curves and improvements on the remaining benchmarks are encouraging, but do not fully resolve these concerns.
Questions for authors
- How does temporal-agreement gating compare with simpler alternatives based on signal magnitude or student confidence? Do you have evidence that the temporal information provides an additional benefit?
- Does the gain mainly come from removing signals with nonpositive agreement, or from downweighting partially aligned signals? Can these two effects be tested separately?
- How many independent training runs were performed? Can you report variation across runs and paired, problem-level confidence intervals for the main improvements?
- Since AIME24 is used for checkpoint selection, do the improvements hold when it is excluded from the test average? Do they also hold with a fixed final checkpoint or a separate validation set?