acceptodds
Under review as a conference paper at ICLR 2027

THINKING FAST WHEN YOU HAVE NO TIME: TRAINING REASONING LLMs UNDER TIME CONSTRAINTS

Abstract

Reasoning language models achieve strong accuracy by generating long chain- of-thought (CoT) traces, but these traces dominate inference cost. Prior work reduces this cost using explicit mechanisms such as length penalties, early-exit mechanisms, or distilled shorter traces that constrain output length. In this work, we hypothesize that shorter responses can emerge implicitly from the structure of the training environment by grouping multiple questions into a single prompt together under a shared response budget (measured in response tokens) without any explicit length-based objective. We test this hypothesis by post-training lan- guage models using reinforcement learning to solve multi-question quizzes. We construct GSM8K-Quiz, Math-Quiz, and SVAMP-Quiz by grouping questions from established datasets into five-question prompts. Crucially, the budget is not uniformly enforced for each question, so the policy is free to spend more reasoning on harder questions and less on easier ones without any explicit difficulty super- vision. We train Qwen2.5-Math-1.5B-Instruct with proximal policy optimization (PPO), separately on GSM8K-Quiz and Math-Quiz. In quiz evaluation, under shared response budgets, the quiz-trained PPO policies use about 50% fewer tokens than the base instruct model on GSM8K-Quiz and 35% fewer on Math-Quiz, with comparable accuracy. In standard single-question evaluation, the efficiency gains persist: on the GSM8K test set, quiz-trained policies use 9–10% fewer tokens than PPO policies trained on single questions, again with comparable accuracy. Our results demonstrate that training in an environment with implicit time pressure can be sufficient to induce compute-efficient reasoning strategies.

Primary area:
reinforcement learning
Keywords:
Large language models, reinforcement learning, efficient reasoning, adaptive computation, Budget-constrained reasoning, mathematical reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_v4w3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes quiz-format RL, which packs five math questions into a single prompt under one shared response-token budget and rewards the model only for the number of correct answers, with no explicit length penalty. The idea is that competition for a shared budget implicitly pushes the model to reason more efficiently and allocate more tokens to harder questions. Training Qwen2.5-Math-1.5B-Instruct with PPO and GRPO, quiz-trained policies use roughly 30–50% fewer tokens than single-question-trained policies on quiz evaluation with comparable or higher accuracy, and the savings partly transfer to single-question benchmarks (about 9–10% fewer tokens on GSM8K).

Critical strengths
  1. Simple idea with minimal changes, easy to adopt. Only the training environment changes (five questions packed together under one shared budget). The reward only counts correct answers, with no length penalty and no difficulty labels, and PPO and GRPO themselves are left untouched. It can be dropped into any existing RL pipeline.
  2. Multiple base models. On DeepSeek-R1-Distill-Qwen-1.5B and Llama-3.2-3B-Instruct, PPO-Quiz cuts quiz response length by more than half with similar or higher accuracy (Table H.1).
Critical weaknesses
  1. The title claimed "THINKING FAST WHEN YOU HAVE NO TIME", so the motivation is related to some concept related to time pressure or let the model aware the time constraint, or at least the model could change the strategy based on the time limit, but the content didn't show any time-related experiments, I think the title is overclaimed.
  2. The baselines in quiz evaluation mostly fail because they run out of tokens. Single-trained policies never saw the quiz format, so they spend 220+ tokens on Q1 and run out by Q5. The base model gives a boxed Q5 answer only 50.2% of the time, versus 98.5% for PPO-Quiz (Table I.2), and 200 of its wrong answers have no box at all, versus 10 for PPO-Quiz (Fig. 6). Meanwhile PPO-Quiz is actually worse on early questions (Q1: 77.9% vs. 84.0% for the base model; Q2: 68.4% vs. 77.2%). So a big chunk of the quiz gain is better coverage under truncation, not better reasoning, which is consistent with Table I.3, where most recovered questions are ones the base model can already solve individually.
  3. The setting is far from the regime the paper is motivated by. Models are 1.5B–3B, the context is 4,096 tokens, and training budgets are 512–1,024 tokens for five questions. This is short-answer math, not the thousands-of-tokens reasoning traces the introduction talks about.
Questions for authors
  1. Some standard deviations in Table 1 are very small, e.g., 63.90 ± 0.14 for GRPO-Quiz and 57.30 ± 0.14 for L1-max on Math-Quiz. Could you report the per-seed results to confirm these values?
  2. On GSM8K-Quiz, PPO-Quiz is less accurate than the base model on Q1 and Q2 (77.9% vs. 84.0% and 68.4% vs. 77.2%, Table I.2). How much of the overall quiz gain comes from better coverage of later questions under truncation versus better reasoning?
  3. Difficulty labels are based on the base model’s solve rate, not the evaluated policy’s. Do the hardness slopes in Table B.1 still hold if difficulty is measured using PPO-Quiz’s own solve rate?
Reviewer_s8bYOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper trains reasoning models on quizzes in which multiple questions share a limited generation-token budget. Through reinforcement learning, the model learns to answer questions within this shared constraint. The paper evaluates accuracy and token efficiency, examines difficulty-sensitive allocation, and studies generalization to individual questions and different quiz sizes.

Critical strengths
  • Training multiple questions under a shared budget is an interesting alternative to imposing separate constraints on each question. The reported improvements in accuracy and token efficiency demonstrate the practical value of this setting and motivate further study of cross-question resource allocation.
Critical weaknesses
  • While the paper reports both aggregate and difficulty-stratified accuracy and token usage, the analysis of how cross-question allocation contributes to these gains remains primarily descriptive. Since the base model already exhibits difficulty-sensitive token usage, the current evidence does not sufficiently isolate improvements in allocation from other benefits of training.
Questions for authors
  • Could the authors clarify how much of the accuracy–efficiency improvement can be attributed to better cross-question allocation, rather than other effects of training? An analysis connecting changes in allocation to changes in accuracy under matched conditions would help establish this link.

  • Beyond the existing fixed-size training and cross-size evaluations, have the authors considered mixing quiz sizes during training, for example, 1, 5, and 10 questions? With training question counts and budget policies controlled, it would be interesting to see whether this further improves the accuracy–token trade-off for both individual questions and quizzes of different sizes; I view this as a potentially valuable extension rather than a prerequisite for acceptance.

Reviewer_1BMTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper shows how to train LLMs to reason more efficiently without length penalties or difficulty labels. The authors introduce quiz-format reinforcement learning, where multiple math problems share a single response budget, creating implicit time pressure that encourages strategic token allocation—spending more tokens on hard problems and fewer on easy ones. Training Qwen2.5-Math-1.5B-Instruct with PPO and GRPO on these bundled quizzes reduced token usage by 35-50% while maintaining or improving accuracy. The efficiency gains generalized to single-question evaluations and new datasets, with experiments confirming the models learned to allocate tokens based on problem difficulty.

Critical strengths

S1. The proposed approach is simple yet effective: it groups multiple questions under a shared response budget during training, without modifying model architecture or optimization. Despite this simplicity, it achieves dramatic efficiency gains—using 35%-50% fewer tokens while matching or exceeding baseline accuracy across multiple datasets.

S2. The evaluation systematically investigate underlying mechanisms and failure modes, making the empirical findings more credible and practical.

Critical weaknesses

W1. The observed budget-sharing advantage may arise from reduced per-question token limits rather than adaptive allocation. As PPO-Single and PPO-Quiz (5 problems per budget) share identical total budgets, Quiz allocates one-fifth per question while Single uses the full budget. Without a control imposing comparable per-question limits on single-question training, it remains unclear whether gains reflect allocation strategy or simply tighter token constraints.

Questions for authors

Q1. Does Quiz outperform Single training when each Single question receives one-fifth the quiz budget independently? This distinguishes whether the benefit arises from budget sharing across questions or from imposing a stricter average limit.

1 comment
AuthorsOpenReview ↗

Response to W1 and Q1: Shared Budget vs. Stricter Per-Question Limits

We thank the reviewer for the encouraging assessment of our approach's simplicity and effectiveness, and for recognizing the systematic analysis of its underlying mechanisms and failure modes. We appreciate the suggestion to distinguish the benefits of shared-budget training from those of stricter per-question token constraints.

We conducted additional experiments to directly address this concern. Specifically, we trained a new PPO-Single control with independent per-question training caps of 102 tokens (GSM8K) and 204 tokens (MATH12K), corresponding to one-fifth of the respective quiz-training budgets, while keeping the quiz evaluation protocol unchanged.

PPO-Quiz outperforms this stricter PPO-Single control, achieving 71.25% vs. 68.29% on GSM8K-Quiz and 59.40% vs. 38.60% on Math-Quiz.

Policy (seed 42)GSM8K-Quiz (Acc. / Tokens)Math-Quiz (Acc. / Tokens)
PPO-Single (full cap)68.21% / 88253.60% / 1,310
PPO-Single (1/5 cap)68.29% / 92238.60% / 668
PPO-Quiz71.25% / 43759.40% / 949

We also examined how frequently the models reached the shared evaluation budget on GSM8K-Quiz. The stricter PPO-Single control (1/5 cap) reached the 1,024-token evaluation cap on approximately 38% of quizzes, compared with 1% for PPO-Quiz. On GSM8K-Quiz, the stricter control achieves nearly identical accuracy to full-cap PPO-Single but still uses substantially more tokens than PPO-Quiz. On Math-Quiz, stricter per-question training caps reduce token consumption but substantially decrease accuracy.

These results show that tighter independent per-question limits alone do not reproduce PPO-Quiz's accuracy–efficiency trade-off, supporting the benefit of shared-budget training.

Importantly, PPO-Quiz does not assign one-fifth of the budget to each question. Instead, all five questions share a single response budget, allowing flexible token allocation across questions without fixed per-question limits.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.