- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies an interesting problem for RL, and from the writing, in particular to RL post training of LLMs where we either provide process rewards to weight each individual choice but sometimes can be wrongly rewarded, or using a terminal reward that only judge the outcome. The author found out that, under a particular problem setting, inaccurate process reward can still be useful in a small data regime but outcome reward wins (smaller minimax regret) when compute budget is larger.
Critical strengths
-
The paper carefully distinguishes known and unknown dynamics and provides consistent results for its claim. I find the analysis under different scenarios helpful and comprehensive.
-
Theorem 4 provides a clean characterization of the loss.
-
The synthetic experiments are consistent with the analytical predictions.
Critical weaknesses
- In the known-dynamics family, the process learner can therefore select its minimax policy before collecting any downstream trajectories. The outcome learner receives fresh information about the true reward and can eventually correct its uncertainty. This is a valid comparison but I am not sure how important this is as the outcome learner eventually winning seems obvious. The construction can thus be restated as a stochastic linear bandit.
- Much of the theory applies to generic objectives. Theorem 3 follows by adding and subtracting proxy values and bounding the span of the errors. Corollary 5 then scales this error by the hybrid weight without characterization. Theorem 4 is more substantive, but its characterization follows from convex duality over occupancy differences.
- Finally I do not understand the comparison with STARC and fail to understand its significance as the paper is too cryptic on such concepts.
Questions for authors
- Which conclusion genuinely depends on stepwise PRM? In particular, would replacing the supplied PRM in Theorem 1 with the complete terminal predictor in a stochastic bandit setting change either the minimax risks?
- How should the conditional Bernoulli noise assumption be interpreted in the motivating RLVR setting? Can the authors give detailed characterizations of such assumptions?