- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes quiz-format RL, which packs five math questions into a single prompt under one shared response-token budget and rewards the model only for the number of correct answers, with no explicit length penalty. The idea is that competition for a shared budget implicitly pushes the model to reason more efficiently and allocate more tokens to harder questions. Training Qwen2.5-Math-1.5B-Instruct with PPO and GRPO, quiz-trained policies use roughly 30–50% fewer tokens than single-question-trained policies on quiz evaluation with comparable or higher accuracy, and the savings partly transfer to single-question benchmarks (about 9–10% fewer tokens on GSM8K).
Critical strengths
- Simple idea with minimal changes, easy to adopt. Only the training environment changes (five questions packed together under one shared budget). The reward only counts correct answers, with no length penalty and no difficulty labels, and PPO and GRPO themselves are left untouched. It can be dropped into any existing RL pipeline.
- Multiple base models. On DeepSeek-R1-Distill-Qwen-1.5B and Llama-3.2-3B-Instruct, PPO-Quiz cuts quiz response length by more than half with similar or higher accuracy (Table H.1).
Critical weaknesses
- The title claimed "THINKING FAST WHEN YOU HAVE NO TIME", so the motivation is related to some concept related to time pressure or let the model aware the time constraint, or at least the model could change the strategy based on the time limit, but the content didn't show any time-related experiments, I think the title is overclaimed.
- The baselines in quiz evaluation mostly fail because they run out of tokens. Single-trained policies never saw the quiz format, so they spend 220+ tokens on Q1 and run out by Q5. The base model gives a boxed Q5 answer only 50.2% of the time, versus 98.5% for PPO-Quiz (Table I.2), and 200 of its wrong answers have no box at all, versus 10 for PPO-Quiz (Fig. 6). Meanwhile PPO-Quiz is actually worse on early questions (Q1: 77.9% vs. 84.0% for the base model; Q2: 68.4% vs. 77.2%). So a big chunk of the quiz gain is better coverage under truncation, not better reasoning, which is consistent with Table I.3, where most recovered questions are ones the base model can already solve individually.
- The setting is far from the regime the paper is motivated by. Models are 1.5B–3B, the context is 4,096 tokens, and training budgets are 512–1,024 tokens for five questions. This is short-answer math, not the thousands-of-tokens reasoning traces the introduction talks about.
Questions for authors
- Some standard deviations in Table 1 are very small, e.g., 63.90 ± 0.14 for GRPO-Quiz and 57.30 ± 0.14 for L1-max on Math-Quiz. Could you report the per-seed results to confirm these values?
- On GSM8K-Quiz, PPO-Quiz is less accurate than the base model on Q1 and Q2 (77.9% vs. 84.0% and 68.4% vs. 77.2%, Table I.2). How much of the overall quiz gain comes from better coverage of later questions under truncation versus better reasoning?
- Difficulty labels are based on the base model’s solve rate, not the evaluated policy’s. Do the hardness slopes in Table B.1 still hold if difficulty is measured using PPO-Quiz’s own solve rate?
Response to W1 and Q1: Shared Budget vs. Stricter Per-Question Limits
We thank the reviewer for the encouraging assessment of our approach's simplicity and effectiveness, and for recognizing the systematic analysis of its underlying mechanisms and failure modes. We appreciate the suggestion to distinguish the benefits of shared-budget training from those of stricter per-question token constraints.
We conducted additional experiments to directly address this concern. Specifically, we trained a new PPO-Single control with independent per-question training caps of 102 tokens (GSM8K) and 204 tokens (MATH12K), corresponding to one-fifth of the respective quiz-training budgets, while keeping the quiz evaluation protocol unchanged.
PPO-Quiz outperforms this stricter PPO-Single control, achieving 71.25% vs. 68.29% on GSM8K-Quiz and 59.40% vs. 38.60% on Math-Quiz.
We also examined how frequently the models reached the shared evaluation budget on GSM8K-Quiz. The stricter PPO-Single control (1/5 cap) reached the 1,024-token evaluation cap on approximately 38% of quizzes, compared with 1% for PPO-Quiz. On GSM8K-Quiz, the stricter control achieves nearly identical accuracy to full-cap PPO-Single but still uses substantially more tokens than PPO-Quiz. On Math-Quiz, stricter per-question training caps reduce token consumption but substantially decrease accuracy.
These results show that tighter independent per-question limits alone do not reproduce PPO-Quiz's accuracy–efficiency trade-off, supporting the benefit of shared-budget training.
Importantly, PPO-Quiz does not assign one-fifth of the budget to each question. Instead, all five questions share a single response budget, allowing flexible token allocation across questions without fixed per-question limits.