- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces Potential-Anchored Consistency of Transitions (PACT), to address the issue of trajectory blindness. The method don't need any sequence-likelihood estimate and importance-ratio clipping. PACT jointly learns the policy and a scalar potential that scores each intermedia state, anchored by terminal reward. The results show large improvement over math and reasoning datasets.
Critical strengths
- Method is simple and straightforward. PACT is likelihood-free and only uses the exact token conditionals of the transitions the sampler actually took. So it avoids the ELBO and masked-reconstruction surrogates and ratio clipping. And results show improvement.
Critical weaknesses
- For the experiments, there is only best of n test set with single seed. Also AGRPO trains at length 384 with blocks of 12, but PACT trains at 256/32 and also test is under 256/32. So the baseline has train-eval mismatch which may lead to the lower accuracy for baseline and also gap between the original paper.
- The paper claims that the per-transition credit assignment drives the accuracy gains, but there is only ablation for sweeping hyper parameters. And it didn't quite show too much information. I think some experiment like dropping the consistency term or do analysis on potential term is useful.
Questions for authors
Please check the weakness above.
4 comments
3. W2. Ablations to show that per-transition credit drives the gains
We agree that hyper-parameter sweeps alone do not show this, so we ran the experiment you suggest. We changed one term of PACT's loss (Eq. 5) at a time, on GSM8K, with the same sampler, seed, prompts, rollout budget and learning rate:
| variant | consistency residual | intermediate anchors | accuracy | seconds / round |
|---|---|---|---|---|
| PACT | per transition (K = 16) | ✓ | 88.7 | 41 |
| T1: path-level balance | once per path (exact TB, all 128 steps) | ✓ | 87.8 | 138 |
| T2: no anchors | per transition | ✗ (boundary r/β only) | 81.3 | 43 |
| T3: no residual | ✗ | ✓ | 3.9 (collapsed) | 42 |
| TraFL (Appendix B) | once per completion (ELBO surrogate) | ✗ (free log Z) | 83.4 | 39 |
What each variant does:
- T1: path-level balance (is locality needed?). One residual per path, summed over all 128 steps (exact relative trajectory balance), instead of PACT's 16 sampled per-transition residuals. It learns far more slowly (77.8 vs 85.9 at round 200) and takes 3.4× the time per round (138 s vs 41 s on the same 16 GPUs), because it needs a gradient pass through every denoising step.
- T2: no anchors (are the anchors needed?). The potential is pinned to the reward only at the finished answer. Final accuracy drops 7.4 points (81.3 vs 88.7).
- T3: no residual (is the residual needed?). Anchors only. The policy receives no credit signal and thus collapses.
- TraFL (is per-completion balance enough?). One residual per completed answer, through an ELBO surrogate, with a learned normalizer. It peaks at 83.4 and then declines (Appendix B).
Together: the per-transition residual is what trains the policy (T3), the anchors make its credit accurate (T2), and imposing the condition per transition rather than per path or per completion is what makes it fast and stable (T1, TraFL). The per-transition residual on an anchored potential is therefore what makes the method work, and its locality is what makes it efficient. Each variant's loss and a longer discussion are in Section 5.2.
We hope these additions resolve your concerns, and we would be glad to clarify anything further.
I appreciate the author's response. I am keeping my score.
Thank You
Thank you for taking the time to review our responses and updated experiments. We are glad that our clarifications and additional results have addressed your concerns, and we truly appreciate your positive feedback and for keeping the positive score.
Rebuttal
Thank you for the positive assessment and for the concrete suggestions. We followed them, and the additions are marked in blue in the revision: the experiment you suggest, dropping the consistency term, together with two related controls (Section 5.2), and final-checkpoint results (Appendix E). We respond to each point below.
1. W1a. Providing final checkpoint selection and stable performance on multiple seeds.
You are right that best-of-n selection on the test set needed a check, so we added results that involve no selection.
*Our reproduction with the authors' code, 900 rollout rounds, decoded with the paper's evaluation protocol (as in Table 1 of the paper).
PACT loses 0.4 points between its best and its final checkpoint and remains 5.2 points above the strongest reproduced baseline at the final checkpoint (AGRPO, 83.5). The comparison therefore does not depend on checkpoint selection. These results are in the new Appendix E (in blue).
2. W1b. Clarifying AGRPO's train/eval mismatch
You are right that AGRPO is trained and evaluated under different settings. We explain below how we ran it and where the mismatch comes from.
458f139), with its released hyper-parameters, prompts, data and splits, rewards and answer parser:train.yaml(384 tokens in blocks of 12, random unmasking, LoRA 64, lr 5e-6, G = 8, 24 Monte-Carlo samples);eval.yaml(256 tokens in blocks of 32, confidence order).num_train_epochs: 5and no step count. We setmax_steps = 900so that every method trains on the same rollout budget (900 rounds of 128 completions), which keeps the accuracy and compute comparisons on equal terms.