acceptodds
Under review as a conference paper at ICLR 2027

LIAR: Length-Interval-Based Reward Shaping for Reliable and Accurate Chain-of-Thought Reasoning

Abstract

Large Reasoning Models (LRMs) have achieved remarkable success in complex problem solving through Reinforcement Learning with Verifiable Rewards (RLVR), which encourages them to generate chain-of-thought (CoT) reasoning traces. However, CoT length critically influences performance: excessively long traces increase inference cost and accumulate errors, while overly short ones break logical coherence and degrade accuracy. Although prior work suggests that the optimal CoT length varies with problem difficulty, directly targeting a single optimal length during policy learning remains impractical. In this paper, we propose the (ORI), a statistically grounded interval of near-optimal reasoning lengths that contains the theoretical optimum and maintains high reliability across its range. Building on ORI, we introduce ength-nterval-bsed eward shaping (), which applies a piecewise-linear length penalty to correct trajectories outside a data-driven ORI estimate, thereby guiding reasoning toward a reliable and high-accuracy length regime. Extensive experiments across four base models show that steering generation into a length interval—rather than compressing toward the shortest one—yields higher accuracy than prior CoT compression methods: in our primary DeepSeek-R1-Distill-Qwen-1.5B setting, LIAR improves over the strongest compression baseline LASER-D by \textbf{+1.7\\%} on average, with the largest gains on the hardest benchmarks (\textbf{+2.7\\%} on AIME 2026), where long reasoning is needed to stay reliable.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Chain-of-Thoughts, Reward Shaping

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_PcVSOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes the Optimal Reasoning Interval (ORI) and Length-Interval-based Reward Shaping (LIAR). The authors map success probabilities at a fixed reasoning depth to reliability under repeated sampling, then define the ORI as the set of lengths satisfying a reliability threshold. In training, they group problems by current accuracy, estimate target intervals from the cumulative distribution of the shortest correct monitoring trajectories, and apply piecewise-linear penalties to correct trajectories outside the interval and overly long incorrect trajectories.

The experiments collectively cover four models, eight mathematical tasks, and four out-of-domain tasks. Across the five main mathematical tasks on Qwen-1.5B, LIAR improves average accuracy over LASER-D by 1.7 pp, while average generation length increases from 3740 to 5472 tokens. The paper provides unified evaluation, comparisons under identical generation caps, and an analysis of decoding uncertainty.

Critical strengths
  1. Clear presentation and reasonably broad experimental coverage. The authors evaluate interval-based shaping across multiple models and tasks beyond mathematics, with unified sampling, some component ablations, and decoding bootstrap analysis. Average accuracy gains over LASER-D across the four models are 1.7/2.5/1.2/1.8 pp, and the main-table averages can be reproduced from the reported entries, supporting the method's empirical value.

  2. The ORI formalization and endpoint rules offer identifiable methodological contributions. The authors construct a cumulative curve from whether each problem has a correct trajectory within a given budget, then determine the two endpoints using a success-rate threshold and residual success mass.

Critical weaknesses
  1. The guarantees for the estimated intervals have gaps. Proposition D.4(ii) overlooks the constraint L≥L_l in Eq.2: kη≤ε does not rule out L_l exceeding the tail cutoff, so it cannot guarantee that L_r stays below that cutoff. Eq.4 on p.21 also contains an algebraic error. The 56 unsuccessful longest samples in Appendix E do not establish the per-query condition η≤0.625% required for the high-difficulty group. Guarantees for the theoretical ORI, empirical cumulative curve, and reward interval need to be distinguished. Additional assumptions or narrower claims may resolve these issues; they do not directly invalidate the main-table results.

  2. Cost comparisons and ablations do not isolate the contribution. LIAR gains 1.7/2.5 pp over LASER-D on 1.5B/7B while using approximately 46%/56% more tokens; Table 7 only matches caps. ARLCP (ICLR 2026) uses the same backbone family and DeepScaleR data and reports higher mean accuracy than LASER in its own evaluation, yet is neither cited nor compared under a common protocol. Setting L_r=L_max in Table 6 also changes L_0 and the overlength penalties for correct and incorrect trajectories; removing tiers also unifies τ. Controls isolating the penalty on short correct trajectories, online updates, and new endpoint rules are missing, as are training-run uncertainty estimates for Table 6.

  3. Related work and citations need fuller coverage. Relevant precedents include ALP's continuous penalty adjustment based on online accuracy, LAPO's quantile intervals and two-sided rewards, DA-DLER's difficulty-grouped truncation, and DRER's dynamic interval-based advantage adjustment. ALP, DRER, and Video-R1 are not cited, while the relevant mechanisms in LAPO and DA-DLER are insufficiently discussed. Accuracy-based difficulty adaptation and interval rewards should not themselves be presented as new contributions. Minor corrections include classifying AutoThink and Thinkless as adaptive reasoning methods learned through training, and citing the original LiveCodeBench and DeepSeekMath, which introduced GRPO.

Questions for authors
  1. Theoretical guarantees: Please correct D.4(ii) and Eq.4, distinguish the guarantees for the theoretical ORI, empirical estimates, and intervals after stabilizer updates, and explain how Eq.3 handles selection effects from grouping the same rollouts used for estimation.

  2. Cost and ablations: Can you compare with LASER-D at similar actual token usage and report monitoring overhead? Please isolate the penalty on short correct trajectories and compare dynamic endpoints with frozen or simple quantile intervals, controlling the penalty scale and incorrect-trajectory branch. Clarify how ε is set without tiers and report training-run variability for key ablations.

  3. Baselines and related work: Can you include ARLCP accuracy and token comparisons under a common protocol, or explain specific factors that prevent a comparable evaluation? Please clarify the contribution beyond ALP's accuracy-adaptive penalties and LAPO/DRER's interval mechanisms, and add the relevant citations and discussion.

Correcting the theory and providing key controls at comparable cost could change my score.

5 comments
AuthorsOpenReview ↗

Response to Reviewer PcVS

We thank the reviewer for the careful and constructive comments. We respond to each comment as follows.


W1 & Q1. The defective bound, Eq. 4, and what each object guarantees

1. The statement. Proposition D.4(ii) overlooked L ≥ L_l: the right-endpoint rule minimises over a feasible set whose lower bound is L_l, so k η_D ≤ ε_D alone does not keep L_r below the tail cutoff. The revised proposition carries both premises — (a) k η_D ≤ ε_D and (b) S_D(L_max) ≥ τ_D + ε_D — and concludes L_r ≤ γ_θ n̂_D, hence L_l ≤ γ_θ n̂_D since L_l ≤ L_r by construction. Premise (b) cannot be dropped: a remark gives a configuration satisfying (a) in which L_l already lies past the cutoff, so the rule returns L_r = L_l.

2. Eq. 4. The decomposition did contain an algebraic error; the correct tail term is (1−p_q(L))^k [1 − (1 − x_q(L)/(1−p_q(L)))^k], from 1 − p_q(L_max) = (1 − p_q(L)) − x_q(L). Since 1 − (1−u)^k ≤ k u, the chain k x (1−p)^{k−1} ≤ k x and every downstream statement are unchanged.

3. What each object carries. (i) The theoretical ORI is a depth-space object; we do not estimate it and claim no recovery result. (ii) The curve S_D is an estimator conditional on the current policy, with a curve-level band rather than pointwise intervals. (iii) The freshly estimated interval is the only object carrying a bound: S_D(L_r) ≥ S_D(L_max) − ε_D holds unconditionally by construction, while L_r ≤ γ_θ n̂_D needs both premises. (iv) The interval used in training is the stabilized one, and the bound does not transfer: the stabilizer delays contraction but never accelerates it, so only the direction is guaranteed. Appendix D tabulates what is established for each.

4. Eq. 3. It is a conditional identity for a fixed monitoring batch, S_D(L) = (1/|D^M_D|) Σ_q [1 − (1 − p_q(L))^k]; we never estimate p_q(L)---we count each question's earliest correct rollout. The same k rollouts tier a question and build the curve, so the statistic and the quantity it defines share sampling noise, and we do not claim to have removed that coupling. Three things keep the procedure well defined: the endpoint rule is a deterministic functional with no hypothesis test; all guarantees are conditional on the batch, so endpoints are operating parameters rather than population estimates; and the monitoring set is disjoint from the gradient minibatches and supplies no reward samples.

5. The evidence. A zero count over 56 rollouts does not resolve a tail rate down to η_D ≤ ε_D/k. We did not relax the tolerance; we rebuilt the test.

Table 1 — the rebuilt tail test, LIAR 1.5B, 5,001 questions, k = 8. n̂_D = tier cutoff in tokens; n_tail = rollouts at or beyond it; ok = the correct ones; CP95 = one-sided 95% Clopper–Pearson bound on the tail correct rate; budget = ε_D / k.

TierQuestionsn̂_Dn_tailokη̂ (%)CP95 (%)Budget (%)
low1,6678,00050620.3951.2391.250
medium1,66712,0001,81470.3860.7240.875
high1,66716,0003,642120.3290.5330.625

All three tiers meet their budget. Cutoffs are fixed per tier by a length-only criterion before correctness is seen, and cap hits count as failures.

W2 & Q2. Cost, the ablation table, and the controls

1. Cost. The arithmetic is right and it settles the claim it tests: 5,472 / 3,740 = +46.3% on the 1.5B pair and 6,006 / 3,855 = +55.8% on the 7B pair. §5.2 of the submitted version described the method as reaching comparable accuracy at lower cost; the revision withdraws that description. In its place Appendix H reports 24 accuracy–cost points — two models, three benchmarks, four budgets, avg@16 throughout — with measured generated tokens on the x-axis. A shared cap constrains generation without equalising it, so the submitted table is not a cost comparison, and we report no interpolated points: adjacent cap settings overlap by 283 tokens on MATH500, by 92 on OlympiadBench, and not at all on AIME 2026. We did not retrain at matched realized cost, and latency is not compared.

2. Monitoring overhead. k × |D^M| = 8 × 500 = 4,000 sequences per endpoint update against 20 × 128 × G = 20,480 between updates, i.e. 19.5% of the generation budget, independent of the training horizon. That is a share of generation, not of total training cost.

3. The ablation table. The rows of the submitted Table 6 bundle changes: setting L_r = L_max also moves L_0 = λ L_r and rescales the overlength penalties on both branches, and removing the tiers collapses three τ_D and three ε_D into single values. Neither row isolates the factor in its label, so the sentence in §5.2 attributing the no-tiers drop to difficulty adaptivity is withdrawn and the rows are reported as component sensitivities. The same applies to ε: the value used there was not recorded in a reconstructable form, so the row is a joint intervention, and the rule to use is tier-size-weighted, ε = Σ_D |D^M_D| ε_D / Σ_D |D^M_D|, which is 0.0771 on the 233/94/173 sizes.

4. Controls. Three variants, each changing one component with tiers, τ_D, the online curve, the penalty scale and the update schedule held fixed.

Table 2 — single-factor component ablations (1.5B). Δ is relative to LIAR under the same evaluation protocol; the parenthesised change is in generated tokens against LIAR's 5,472.

VariantChangeMacro acc (%)Avg TokΔ (pp)
LIAR—50.55,472—
Adaptive single pointL_l = L_r, tiers and online curve kept49.16,317 (+15%)−1.4
Frozen intervalendpoints fixed before training49.86,196 (+13%)−0.7
Upper boundary onlyshort-correct penalty removed49.04,244 (−22%)−1.5

The three rows ask, in order, whether two endpoints are needed, whether they must be re-estimated, and whether the short-correct penalty contributes — removing it saves a fifth of the tokens at a cost of 1.5 points. Every removal costs accuracy, and the token directions differ: interval variants +13–15%, short-correct penalty −22%. A LAPO-style quantile interval was not run, because it replaces the endpoint definition rather than removing one component, so it is not a control.

5. Variance. The submitted table is one training run. We add one more seed of the same recipe (s1) to the one used in the paper (s0), so the comparison rests on two LIAR seeds against the frozen LASER-D checkpoint.

Table 3 — seeds and question-level intervals. Avg Tok is the macro average generated length; the CI is a two-level hierarchical bootstrap (questions, then 16 decoding samples) on the macro gap against LASER-D.

RunMacro acc (%)Macro Avg TokTwo-level 95% CI (macro Δ)
s050.505,472[+0.9, +2.7]
s150.505,361[+0.7, +2.5]
LASER-D (frozen, one checkpoint)48.83,740—

Both seeds agree on macro accuracy and differ by 111 tokens in macro length; the macro lower bound is positive for both. With the baseline frozen, this is a one-sided lower bound on total variability.

ARLCP is answered with Question 3 below.

W3 & Q3. Prior work, novelty, and ARLCP

1. What is not new. Difficulty adaptation from online accuracy and rewards shaped on an interval of lengths are both prior art; the introduction now says so and places the contribution elsewhere. We have added ALP, DRER, Video-R1, the original LiveCodeBench paper, DeepSeekMath, and ARLCP (ICLR 2026), and corrected AutoThink and Thinkless to adaptive reasoning methods learned through training.

2. What the contribution is. (i) The endpoints come from a tail condition on the success-ratio curve — L_l is the first budget reaching τ_D, L_r the first after which further depth adds less than ε_D of success mass — not from a length percentile or a difficulty statistic. (ii) The criterion is measurable from one online batch of k = 8 rollouts, needs no question with several correct traces, and stays defined for questions that no rollout solves. (iii) S_D(L_r) ≥ S_D(L_max) − ε_D follows from the selection rule, and L_r ≤ γ_θ n̂_D only under both premises of the corrected proposition.

3. The closest precedent. DRER, not LASER-D: it is already bucket-wise, two-sided and online. The targets differ in what they are statistics of — percentiles of the lengths of correct traces within a bucket (DRER P5/P95; LAPO P30/P70 per question) versus the tail of the success-ratio curve with a declared tolerable loss ε_D. ALP adapts a penalty to online accuracy, DA-DLER groups truncation by difficulty, ARLCP estimates complexity; none places endpoints this way.

4. ARLCP. No official checkpoint is public, and the published results do not cover all five benchmarks under our protocol, so we train a 1.5B model to the same recipe and evaluate it in our harness.

Table 4 — ARLCP against the two reference points. avg@16, same harness, macro over the five main benchmarks.

ModelMacro acc (%)Macro Avg Tok
LIAR50.505,472
LASER-D48.83,740
ARLCP (1.5B)49.343,401

ARLCP spends 62% of LIAR's tokens and 91% of LASER-D's, with accuracy in between — a third point on the same curve. Appendix K lists the protocol axes: backbone, corpus, evaluation protocol, parser, prompt, sampling, training schedule and training L_max all match; only the reflection detector and its corpus statistics are ARLCP-specific, reproduced from the paper. Axes that differ are marked "not directly comparable on that axis"; we make no equivalence claim to the published numbers.


We sincerely hope that our rebuttal could properly address your concerns. If so, we would deeply appreciate it if you could raise your score. If not, please let us know your further concerns, and we will continue actively responding to your comments and improving our submission.

Reviewer_UpGuOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

LIAR studies how to train reasoning models to produce chains of thought that are long enough to solve difficult problems without wasting tokens on unnecessary continuation. Its central idea is to reward a range of reasoning lengths, rather than continually favoring shorter answers or targeting a single length. The paper defines an Optimal Reasoning Interval through a probabilistic model of correctness, then introduces a practical approximation: periodically sample solutions on held-out monitoring questions, group questions by observed difficulty, and estimate length intervals from the fraction of questions with a successful solution within each token budget. For the primary 1.5B model, the paper shows LIAR can obtai more accurate and shorter responses than the original unmodified model.

Critical strengths

Well-motivated, practical approach to balancing accuracy and reasoning cost. The interval reward incorporates variation in useful reasoning length and adapts to observed difficulty level. It retains a preference for correct answers while discouraging both excessive compression and unnecessary continuation.

Good experiment support The reported average accuracy gains appear across four models, including Qwen and Llama-based models. Ablations provide some evidence that the design matters: removing difficulty tiers lowers accuracy by 1.2 percentage points, while removing the upper endpoint increases token use by 32% and lowers accuracy by 0.7 points.

Critical weaknesses

The theoretical framing is stronger than the connection established in the actual method. The formal ORI focuses on correctness at a particular reasoning depth, whereas the practical estimator measures cumulative success within a certain token budget. That statistic also depends on which lengths the policy generates, so its plateau need not to identify a reliable reasoning interval.

Questions for authors
  • How does Eq. 3 account for empirical estimation and outcome-dependent tier selection?
  • The budget sweep in Table 7 uses the same token caps, but LIAR generates more tokens than LASER-D in every row. Can you compare accuracy at similar actual average token usage using the existing sweep, and report variation across training seeds if available? This would help clarify the trade-off and robustness of the gains.
4 comments
AuthorsOpenReview ↗

Response to Reviewer UpGu

We thank the reviewer for the positive and insightful comments. We respond to each comment as follows.


W1. The theoretical framing is stronger than the connection the method establishes

You are right, and the reason is sharper than a missing experiment. S_D(L) is a cumulative success statistic over the lengths the current policy generates; ORI is a level set of pointwise reliability R(N;k,C) in depth space. A plateau in S_D need not be that level set: the increment vanishes when deep traces are inaccurate, when such depths are rarely proposed, or when the tier is already saturated, and only the first is the decline the endpoint rule tracks (Appendix D).

Two things are settled and one is not. Settled: ORI(k,C,τ) is defined as the level set of R, and the inclusion corollary of Section 3 places the optimal length N* inside it for any τ ≤ max_N R(N;k,C) — a result about ORI, independent of our estimator. Also settled: under Assumption 2.1, [L_l, L_r] is a first-order image of ORI, and Prop. D.4(ii) bounds the upper side only — L_r ≤ γ_θ n̂_D when both premises hold — so within those premises the interval does not run into the over-reasoning regime. Not settled: that the image is close — we prove no lower bound on L_l and no L_r ≥ γ_θ n_r, hence no two-sided coverage, and we do not present the surrogate as containing ORI's image.

Appendix D now separates the four objects: the depth-space ORI (not estimated), the empirical curve, the fresh interval (the only one carrying a bound), and the stabilized interval (no bound transferred). The Limitations name the two gaps this leaves: recovering the theoretical ORI from a token-budget interval, and aligning the two under an identifiable length–depth map.

Q1. How Eq. 3 accounts for empirical estimation and outcome-dependent tier selection

Eq. 3 is a conditional identity, not an estimation equation. In the implementation S_D(L) is a count: for each monitoring query we take the length of its earliest correct rollout, if any, and S_D(L) is the fraction of queries in the tier whose earliest correct length is at most L, read at each grid point up to L_max. p_q(L) is notation for that identity and is never estimated. Because the curve comes from one batch and neighbouring budgets are strongly correlated, its uncertainty is a curve-level band rather than independent pointwise intervals.

Tier assignment uses the same k rollouts as the curve, so the statistic and the quantity it defines share sampling noise, and we do not claim to have removed that coupling: endpoint values are conditional on this draw, not exchangeable with what a fresh batch would give. Three things keep the procedure well defined. The endpoint rule is a deterministic functional of the realized curve with no hypothesis test, so the estimator stays well defined under the coupling. L_l and L_r are therefore presented as operating parameters of the reward, not as unbiased estimates of a population quantity. And the monitoring set is disjoint from the GRPO minibatches and supplies no reward samples: its only product is the endpoint pair, and every reported accuracy comes from an independent avg@16 evaluation.

Q2. Matched realized-token comparison and variation across training seeds

We agree on the accounting: a shared cap does not imply matched actual token usage, and the submitted shared-cap sweep showed LIAR generating more tokens than LASER-D in every row; that table is replaced in the revision by the curves described below. Rather than interpolate the old table, we provide a full accuracy–cost curve in Appendix H: 24 points, 2 models × 3 benchmarks × 4 budgets, evaluated with avg@16 under the same protocol as the main table, with measured average generated tokens on the x-axis. The old table cannot yield this curve: its overlap band is 1,562–1,845 tokens on MATH500 (3 evaluable points), 3,321–3,413 on OlympiadBench (2 points), and there is no overlap at all on AIME 2026 (LIAR's shortest 6,499 vs LASER-D's longest 5,519). The figure is in Appendix H; we make no Pareto-dominance claim, and latency is not compared.

Seeds. The main-table comparison contains the seed used in the paper and one additional LIAR run of the same recipe (macro accuracy 50.50 and 50.50, average generated length 5,472 and 5,361 tokens), against the single released LASER-D checkpoint, so it is a one-sided lower bound on training variance rather than an estimate of it. We did not run the sweep for a second seed: each point is a dedicated budget-forcing run with 16 samples per question, which could not be completed within the discussion period.


We sincerely hope that our rebuttal could properly address your concerns. If so, we would deeply appreciate it if you could raise your score. If not, please let us know your further concerns, and we will continue actively responding to your comments and improving our submission.

AuthorsOpenReview ↗

Kindly Reminder

As the discussion period draws to a close, we would like to sincerely thank you again for your thoughtful comments and valuable feedback. We greatly appreciate the time and effort you have devoted to reviewing our work, and we hope that our rebuttal and additional evidence have helped address your concerns. We would be grateful if you could take these clarifications into account in your final assessment.

Thank you again for your time and consideration.

Reviewer_UpGuOpenReview ↗

Thank you for your careful rebuttal

Thank the author for the careful rebuttal and most of my questions are addressed. I already gave a weak acceptance score and would like to maintain this position. Yet I have adjusted up my confidence score as I get more familiar with the topic.

We sincerely thank you for your time across two rounds, for keeping your assessment of our work, and for raising your confidence in it.

The four-object separation in Appendix D, the measured accuracy–cost curves, and the seed-level evidence all grew out of your questions, and both the paper and our understanding of our guarantees are substantially better for them.

Thank you again for a review whose questions made the revision stronger.

Reviewer_AytHOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LIAR, a reward-shaping method for reinforcement learning with verifiable rewards intended to avoid both under- and over-reasoning. The paper first defines an Optimal Reasoning Interval (ORI) as a superlevel set of a binomial rollout-reliability function induced by fixed-depth accuracy. The practical algorithm does not estimate that function directly: on a 500-query monitoring set, it partitions queries into three online difficulty tiers using k = 8 rollout correctness, forms a cumulative success-ratio curve SD(L)—the fraction of queries having at least one correct rollout of length at most L—takes a threshold crossing as L_l and a tail-saturation point as L_r, and applies a bounded piecewise-linear penalty outside [L_l,L_r]. Correct traces are penalized for being too short or too long; incorrect traces are penalized only for excessive length. The main empirical result is higher average accuracy than the strongest reported compression baseline, LASER-D, across four trained model settings. In the primary 1.5B setting, LIAR improves macro accuracy by 1.7 percentage points while using 46% more tokens than LASER-D, though substantially fewer than the uncompressed model. The paper also reports additional mathematical benchmarks, four out-of-domain tasks, hyperparameter sweeps, a shared-cap decoding study, and conditional theoretical results linking the cumulative estimator to rollout reliability.

Critical strengths
  1. Simple and useful reward design. The bounded penalty preserves correctness as the primary objective: every correct trajectory scores above every incorrect trajectory, while the interval gives successful traces flexibility instead of enforcing one exact length.
  2. Transparent accuracy-cost reporting. Accuracy and average length are reported together, and the authors explicitly avoid claiming Pareto dominance where LIAR is more accurate but longer.
  3. Evidence of a real hard-task effect. The largest gains repeatedly occur on difficult AIME benchmarks across model settings. This is consistent with the hypothesis that aggressive compression can under-allocate reasoning, although it does not prove the ORI mechanism.
  4. Strong supplementary material. The hyperparameter sweeps, proofs, contamination check, additional models, and explicit limitations are materially useful and unusually candid.
Critical weaknesses
  1. The implemented interval is not shown to estimate the theoretical ORI. ORI is defined from reliability at an exact depth; LIAR uses a cumulative budgeted pass@8 statistic marginalized over the model’s length distribution. The paper proves only limited one-sided properties of the right endpoint and no recovery or approximation result for either ORI boundary.
  2. The central reliability assumption is weakly validated. The tail-collapse check uses one benchmark, one post-training model, and only 17 mixed-outcome questions. Aggregate within-question quintiles do not verify the stated uniform per-query condition, and no evidence is provided over training, across tiers, or across models.
  3. The length-depth assumption receives only indirect support. It is tested using text-derived step proxies that are mechanically correlated with length. The reported correlations are useful diagnostics but do not strongly validate latent semantic reasoning depth.
  4. The key causal ablation is missing. LIAR is not compared against an otherwise identical adaptive single-point target, fixed interval, upper-only reward, lower-only reward, or a version using the same online tiers and success curve without an interval. Gains relative to LASER-D therefore cannot be attributed specifically to interval targeting rather than tiering, reward scaling, increased length, or other recipe differences.
  5. Compute matching is inadequate. A shared maximum cap is not a matched realized-token or matched-latency comparison. On easy and medium tasks, LIAR is often tied or slightly worse at shorter caps. Frontier claims should be based on models tuned to comparable realized cost or on full accuracy-cost curves.
  6. Statistical evidence omits the largest sources of uncertainty. The bootstrap fixes benchmark questions, apparently pairs arbitrary decoding positions, and excludes training seeds. It measures within-checkpoint decoding variation but cannot establish that a 1–2 point gain will reproduce across training runs or question samples.
  7. Reproducibility and efficiency accounting are incomplete. Core optimizer details, seed counts, exact baseline provenance, and percentage overhead from the 500×8 monitoring rollouts every 20 steps are not fully reported.
Questions for authors
  1. Can the authors directly compare each learned [L_l,L_r] with an independently estimated fixed-length accuracy/reliability curve for the same tier and checkpoint, reporting endpoint error, ORI coverage, and how often the learned interval truncates empirically reliable depths? Without this, the claimed bridge from ORI theory to LIAR remains unverified.
  2. How many independent training seeds were run for LIAR and the strongest baseline? Please report seed-level accuracy and token counts and recompute uncertainty with questions as clusters, or with a hierarchical bootstrap over questions and decoding samples.
  3. Can the authors provide a controlled comparison at matched realized token count or latency, and an ablation in which all training details and online difficulty tiers are held fixed while replacing the interval with (i) a single target and (ii) only one boundary? These results would determine whether the gain comes from interval targeting rather than extra compute or tiering.
  4. The reliability guarantee appears to depend on Assumption D.3. Is this assumption satisfied for each query, or only on average within a difficulty tier? Please provide those measurements together with depth-proposal coverage near the intended interval. If not, which claims about a “reliable interval” survive beyond the unconditional tail-mass statement?
6 comments
AuthorsOpenReview ↗

Response to Reviewer AytH

We thank the reviewer for the careful and constructive comments. We respond to each comment as follows.


W1 & Q1. The implemented interval is not shown to estimate the theoretical ORI

We agree: the submitted paper does not show that the implemented interval estimates the theoretical ORI, and proves no recovery or approximation result for either boundary.

We keep the estimator in token space for four reasons: (i) true reasoning depth is unobserved and text-derived step proxies disagree; (ii) the length–depth slope is itself estimated, so targeting depth would inject its error into the reward; (iii) the quantities controllable during generation are token-level; (iv) an endpoint must be in tokens to serve as a training signal.

The paper states the step–token relation as an explicit assumption in the Preliminaries (L = γ_θ N + o(N), with γ_θ the average tokens per step) and tests it in Appendix F with step proxies N̂; recovering the theoretical ORI and aligning the two spaces are left as future work in the Limitations.

First, a premise correction on your Question 1: LIAR never truncates a rollout. There is no accept/reject step and the accuracy reward is unchanged; a correct trajectory past L_r receives only the bounded Φ₊ penalty (floor −0.5) — a soft length penalty, not truncation.

The comparison you ask for is available in curve form. Endpoints are placed from the monitoring pool D^M (disjoint from policy-gradient minibatches), whereas the length–accuracy relation is measured with avg@16 on evaluation questions that never enter D^M. That independent curve is the 24-point accuracy–cost sweep of Appendix H, on which the LIAR operating points are where the learned endpoints land; a shared cap would not answer it, because a cap is not a realized length. We cannot report ORI coverage: ORI lives in depth space, and these are empirical landing points of the rule, not a verification of the theory–practice bridge.

W2 & Q4. The central reliability assumption is weakly validated

We accept the criticism: the original check used one benchmark, one post-training model, 17 mixed-outcome questions, and within-question quintiles, which cannot verify a uniform per-query condition. We have rebuilt it on the LIAR main checkpoint with 5,001 questions balanced to 1,667 per tier, k = 8 rollouts each (40,008 rollouts), and tier-specific absolute cutoffs of 8,000 / 12,000 / 16,000 tokens, fixed by a length-only power criterion and independent of correctness. 32,768-token cap hits are scored as failures and kept in the denominator.

The check also differs in kind: the tail-collapse assumption bounds a rate (tail accuracy ≤ η_D = ε_D/k), so counting correct long rollouts cannot test it. We therefore report a one-sided 95% Clopper–Pearson upper bound (CP95) against η_D — fixing the cutoff after seeing correctness would be circular — so CP95 > η_D is a counterexample.

Table 1 — main test, LIAR 1.5B, 5,001 questions, k = 8. n̂_D = tier cutoff in tokens; n_tail = rollouts at or beyond it; ok = the correct ones; η̂ = ok / n_tail; CP95 = one-sided 95% Clopper–Pearson upper bound; η_D = the tier budget ε_D / k.

TierQuestionsn̂_Dn_tailokη̂ (%)CP95 (%)η_D (%)
low1,6678,00050620.3951.2391.250
medium1,66712,0001,81470.3860.7240.875
high1,66716,0003,642120.3290.5330.625

All three upper bounds fall at or below their tier budgets; the bounded quantity is each tier's tail rate.

Table 2 — same cutoffs and k = 8 on the released LASER-D checkpoint, 5,001 questions. Same symbols as Table 1.

TierQuestionsn̂_Dn_tailokη̂ (%)CP95 (%)η_D (%)
low1,6678,00035810.2791.3181.250
medium1,66712,0001,11240.3600.8210.875
high1,66716,0002,59670.2700.5060.625

The medium and high tiers meet their budgets on LASER-D; at the low tier the point estimate (0.28%) is within the budget but the CP upper bound (1.32%) exceeds it, because n_tail = 358 with one correct rollout cannot resolve a 1.25% rate — inconclusive, not a violation. It shows the condition is met on a second checkpoint at two of three tiers and not contradicted at the third.

The design also bounds how far the claim reaches, which is your Question 4: Assumption D.3 quantifies over every question in a tier, and we verified it only as a per-tier aggregate. With k = 8 a question contributes at most eight tail samples, so a 0/8 question still carries a 31.2% one-sided bound on its tail error rate against a high-tier budget of 0.625% — a factor of fifty. Per-question certification is therefore out of reach at this k: the fraction of questions meeting η_D would be zero at every tier. Appendix D states the assumption as a sufficient condition, separating four objects of which only the freshly estimated interval carries a bound.

W3. The length–depth assumption receives only indirect support

We agree: a latent semantic quantity cannot be verified from text-derived proxies, and N̂ is counted from the same strings as L, so the fit measures the stability of the token-per-step ratio rather than independently confirming linearity — which is also why the method does not depend on it. Every quantity in our guarantees is measured in tokens — the curve S_D(L), the endpoints L_l and L_r, the tier cutoff n̂_D — and N̂ enters no step of the algorithm, the reward, or any bound. If the linear law failed, the numbers we report would be unchanged; only a depth-based reading of the axis would be affected.

Table 3 — does the step axis add information beyond length? β₁ / β₂ are coefficients on L / on the residual r (r = the part of L not explained by N̂) in a logistic regression for correctness, standard errors clustered by question; LR = likelihood-ratio test with 1 d.f.; AUC = area under the ROC curve.

Settingn / questionsβ₁ (L)β₂ (r)LRAUC(L) → AUC(L, r)
Pooled480 / 30−2.61−1.7217.6 (p<0.001)0.846 → 0.859
Mixed-outcome only240 / 15−2.45−1.567.9 (p=0.005)0.829 → 0.842
Question fixed effects240 / 15—+0.271.95 (p=0.162)—

The residual is significant with questions pooled, but becomes insignificant and changes sign under question fixed effects, and adding r moves AUC by 0.013. Once the question and its length are fixed, the step axis carries no information beyond length, so dropping it costs nothing. This removes a dependence rather than a doubt: we do not claim that length tracks reasoning depth.

W4 & Q3. The key causal ablation is missing

The criticism is correct: we now report three single-factor variants, each changing exactly one component with everything else held fixed, all retrained under the same protocol.

Table 4 — single-factor component ablations (1.5B). Each row changes one ingredient; the rest of the recipe is held fixed. Δ is relative to LIAR; the parenthesised token change is against LIAR's 5,472.

VariantChangeMacro acc (%)Avg TokΔ (pp)
LIAR—50.55,472—
Adaptive single pointL_l = L_r, tiers and online curve kept49.16,317 (+15%)−1.4
Frozen intervalendpoints fixed before training49.86,196 (+13%)−0.7
Upper boundary onlyshort-correct penalty removed49.04,244 (−22%)−1.5

On the length confound, the paper's own main table already shows the gain is not monotone in length: on AIME 2026 the longest model (Original, 17,932 tok) scores 21.2, the shortest (LASER-D, 5,470 tok) 21.5, while LIAR (8,061 tok) and AutoThink (8,863 tok) score 24.2 and 24.0.

Question 3 asks for the same controls and offers a priority ordering: a single target, if only one can be run. That is the first variant row of Table 4; the frozen-interval row was run as well. The “keep tiers, remove the interval” variant was not run, because LIAR and LASER-D already share the same tiering procedure and it is therefore not a new single-factor contrast.

W5 & Q3. Compute matching is inadequate

We agree that a shared maximum cap is not a matched realized-token or matched-latency comparison; the revised Appendix H states this explicitly. We have added a full accuracy–cost curve in Appendix H: 24 points (2 models × 3 benchmarks × 4 thinking-token budgets, same avg@16 protocol and tokenizer), with measured average generated tokens on the x-axis rather than cap labels. We did not interpolate the submitted shared-cap table — its overlap bands are too narrow (283 tokens on MATH500, 92 on OlympiadBench, and no overlap at all on AIME 2026) to support a curve.

Two features of the curve bear on the question. First, the gain is largest where the problems are hardest: on AIME 2026 LIAR is +2.3 pp at the largest budget (23.1 vs 20.8) and +2.7 pp in the main table (24.2 vs 21.5, against 24.0 for AutoThink at 8,863 tokens), whereas the same model gains +1.0 pp on MATH500 (84.8 vs 83.8) and −0.1 pp on OlympiadBench (46.9 vs 47.0). Second, the advantage is not a longer-output effect: at comparable measured cost the LIAR curve already lies above LASER-D's on AIME 2026, and the longest entry in the main table generates far more tokens yet scores below LIAR. More tokens do not by themselves move accuracy there; what LIAR changes is where the tokens go.

We read these as a complete trade-off, not as dominance: no Pareto claim, no retraining at comparable cost, no latency comparison. We also report the crossing rather than only the favourable region: where the cap binds, LIAR is tied or slightly worse while spending more tokens, and OlympiadBench is that case.

This is also our answer to the matched-realized-cost half of Question 3.

W6 & Q2. Statistical evidence omits the largest sources of uncertainty

Question 2 asks whether the intervals carry the two variances that dominate here — question sampling and training-seed sampling — and whether the claim survives them. Both are now addressed, in Table 5 and Table 6.

The first two points are correct about our original protocol: fixing benchmark questions while pairing decoding positions measures within-checkpoint decoding variation only. We have recomputed uncertainty with questions as clusters and with a two-level hierarchical bootstrap (resampling questions, then 16 decoding samples within each question); the resulting intervals, reported in Appendix I, are wider than the paired bootstrap in the submitted paper because they include the question-sampling component.

On training runs, we add one more seed (s1) of the same recipe to the seed used in the paper (s0), so the comparison rests on two LIAR seeds against the frozen, publicly released LASER-D checkpoint. Macro accuracy is 50.50 / 50.50 and macro average generated tokens 5,472 / 5,361, so all five per-benchmark deltas are positive for both seeds.

Table 5 — seeds and question-level intervals. Avg Tok is the macro average generated length; the CI is the two-level bootstrap (questions, then decoding samples) on the macro gap against LASER-D.

RunMacro acc (%)Macro Avg TokTwo-level 95% CI (macro Δ)
s050.505,472[+0.9, +2.7]
s150.505,361[+0.7, +2.5]
LASER-D (frozen, one checkpoint)48.83,740—

Table 6 — two-level hierarchical bootstrap, 95% CI. The two levels are questions, then 16 decoding samples within each question; Δ is LIAR − LASER-D in points against the frozen checkpoint.

SeedBenchmarkQuestionsLIARLASER-DΔ95% CI
s0MATH50050085.684.2+1.4[+0.7, +1.9]
s0AIME 20263024.221.5+2.7[+0.2, +5.8]
s0AMC 20234076.173.8+2.3[+0.1, +5.3]
s0OlympiadBench67548.047.3+0.7[−0.0, +1.6]
s0IMO-Acc-Bench40018.617.2+1.4[+0.1, +2.6]
s0Macro (5)—50.5048.8+1.70[+0.9, +2.7]
s1MATH50050085.884.2+1.6[+0.9, +2.1]
s1AIME 20263024.221.5+2.7[+0.4, +5.9]
s1AMC 20234076.673.8+2.8[+0.3, +5.9]
s1OlympiadBench67547.847.3+0.5[−0.5, +1.1]
s1IMO-Acc-Bench40018.117.2+0.9[−0.5, +2.0]
s1Macro (5)—50.5048.8+1.70[+0.7, +2.5]

The macro lower bound is positive for both seeds, while individual benchmarks with small gains still have intervals containing zero. Because the LASER-D checkpoint is frozen and was not retrained, this is a one-sided lower bound on total uncertainty: we do not claim to account for training-seed variance on the baseline side.

W7. Reproducibility and efficiency accounting is incomplete

This is an accounting gap rather than a missing experiment, and we close it. Monitoring overhead: one endpoint update generates k × |D^M| = 8 × 500 = 4,000 sequences; the 20-step interval between updates generates 20 × 128 prompts × G = 20,480 sequences (G = GRPO group size, 8 rollouts per prompt). The ratio is 4,000 / 20,480 = 19.5%, independent of the total number of training steps. Appendix B gives this arithmetic; wall-clock overhead is smaller, but we do not call it negligible.

Optimizer details: Appendix B now lists the full table: learning rate 1e-6, no LR schedule or warmup override, KL coefficient 0.001 inside the loss (low_var_kl, Decoupled GRPO), asymmetric clip 0.20 / 0.28, training L_max = 16,384 tokens (distinct from the evaluation caps), 5 epochs, batch 128 / mini-batch 64 / micro-batch 2 per GPU, G = 8, temperature 0.6, entropy 0.001, verl.

Baseline provenance: with the exception of DAPO-OS and ARLCP, every baseline is the authors' publicly released checkpoint, evaluated under the same inference framework and the same evaluation protocol as our method (§5.1). DAPO-OS and ARLCP are trained by us on the same backbone and data, under the same protocol. Seed counts: two LIAR seeds against one frozen LASER-D checkpoint (LIAR 2 / LASER-D 1).


We sincerely hope that our rebuttal could properly address your concerns. If so, we would deeply appreciate it if you could raise your score. If not, please let us know your further concerns, and we will continue actively responding to your comments and improving our submission.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their time and constructive feedback. Our main finding is that shaping length rewards toward a reliability interval---rather than toward a single point or a hard cap---improves the accuracy–efficiency trade-off of RLVR-trained reasoning models. LIAR instantiates this by estimating, for each difficulty tier, the token-budget interval over which the current policy stays reliable, and penalizing correct traces only outside that interval.

We have uploaded a revised version of the paper, in which added and revised text is shown in blue. We summarize the revisions below and respond to each reviewer in detail under the corresponding review.

Theory

  • Corrected proposition (Appendix D, summarized in Section 4.2). Proposition D.4(ii) now carries two premises---(a) the tail bound kηD≤ϵDk\eta_D \leq \epsilon_D and (b) tier solvability, SD(Lmax⁡)≥τD+ϵDS_D(L_{\max}) \geq \tau_D + \epsilon_D---and the corrected statement concludes Ll,Lr≤γθn^DL_l, L_r \leq \gamma_\theta \hat n_D; a remark shows that premise (b) cannot be dropped (Reviewer AytH).
  • Corrected Eq. 4 (Appendix G). The tail term of the Eq. 4 decomposition now reads (1−p)k[1−(1−x/(1−p))k](1-p)^k\bigl[1-(1-x/(1-p))^k\bigr]; the bound and all downstream statements are unchanged (Reviewer AytH).
  • Objects and guarantees separated (Appendix D). The depth-space ORI, the success-ratio curve, the freshly estimated interval and the stabilized interval are now stated separately, with the strongest established claim for each; the stabilized interval inherits only the direction of the bound (Reviewers AytH and PcVS).

New experiments

  • Tail-collapse main test (Appendix E, summarized in Section 4.2). 5,001 balanced questions with tier-specific absolute cutoffs and k=8k=8: all three tier-level CP95 upper bounds meet their budgets on the main checkpoint (Reviewer AytH).
  • Cross-checkpoint test (Appendix E). The same test on the released LASER-D checkpoint: the medium and high tiers meet their budgets; the low tier is inconclusive at this sample size (Reviewer AytH).
  • Single-factor mechanism ablations (Appendix H.1, summarized in Section 5.2). Three retrained variants---adaptive single point, frozen interval, upper boundary only---each changing one component with the rest of the recipe held fixed (Reviewers AytH and PcVS).
  • Full accuracy–cost curves (Appendix H.2, summarized in Section 5.2). 24 operating points indexed by measured generated tokens for LIAR and LASER-D; no interpolated points and no dominance claim (Reviewers AytH, UpGu and PcVS).
  • Statistical evidence (Appendix I, summarized in Section 5.2). Two LIAR seeds and a two-level hierarchical bootstrap (questions, then decoding samples) against the released LASER-D checkpoint (Reviewers AytH and UpGu).
  • ARLCP under a common protocol (Appendix, summarized in Section 5.1). A 1.5B ARLCP model trained to the published description and evaluated in our harness, with the protocol axes listed one by one (Reviewer PcVS).

Related work

  • Length-reward statistics and intervals (Appendix). ALP, ARLCP, DRER, LAPO, DA-DLER and Video-R1 are now cited and compared; AutoThink and Thinkless are classified as training-time adaptive-mode methods rather than inference-time interventions (Reviewers AytH and PcVS).

Writing

  • Readability. The abstract, the introduction (including a contributions paragraph) and the opening of Section 4 now state the method and its guarantees directly (Reviewers AytH and PcVS).
  • Claim scope. Section 5.2 no longer describes the method as reaching comparable accuracy at lower cost; accuracy and length are reported as a pair, and comparative claims are stated as measured deltas rather than rankings (Reviewers AytH and PcVS).
  • Compute and monitoring accounting (Appendix B). The monitoring-overhead arithmetic (19.5% of generation rollouts) and the full optimizer table are now given (Reviewer AytH).
  • Unchanged content. The method, the training recipe and all originally reported experimental results are unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.