acceptodds
Withdrawn from ICLR 2027

When Output Budgets Masquerade as Representation Failures in Program-Based Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Comparisons between structured output formats usually give every format the same generation-token cap. When two formats serialize the same computation at different lengths, an equal cap is not an equal opportunity to finish. We show how to identify this confound inside one fine-tuned program representation with paired cross-budget trajectories, and what it hides. In a program-based mathematical-reasoning benchmark whose 512-token evaluation appeared to show flat JSON programs collapsing on five-call problems while compositional plans did not, we re-decode the same three flat checkpoints at 512 and 1536 tokens, with and without JSON-schema constraints, in 24 arms with 19,752 predictions. On the five-call slice the 512-token cap ends 65.7% of skeleton-disjoint and 68.5% of augmented outputs by length; at 1536 tokens no output is cut, parse rates rise from 35.2% and 31.7% to 100%, execution by 60.0 and 63.2 percentage points, and accuracy by 19.0 and 22.8 points. Three tests locate the mechanism: truncating the saved 1536-token trajectories at 512 tokens reproduces the short-cap five-call accuracy exactly and 98.6% of row-level outcomes; exact-prefix pairs carry 247 of the 253 net correct gains; and offline estimates for 640, 768, and 1024 tokens, registered before decoding, match 97.9% of decoded correctness outcomes. Tripling the cap adds 14% generated tokens on the five-call slices. Validity recovers in every held-out five-call skeleton and on all 368 length-finished four- and six-call rows, while the accuracy gain is carried by one skeleton: truncation had turned wrong computations into validity errors. The same censoring appears in every seed on all 1,319 GSM8K test problems, including 206 whose audited reference computation has no training structure, and in a second model family, Llama-3.1-8B-Instruct, with offline truncation predicting at least 98.0% of row outcomes. Executable and semantic audits find 43 test rows whose labels contradict the text and 110 whose quantities contradict a premise of the question; on the rows that pass both, the five-call accuracy gains are 27.8 and 43.7 points. A semantics-preserving inliner shows that the historical converter changed the returned value of 6.0% of training targets. Retraining Qwen3.5-9B on matched targets in both formats and decoding each to saturation leaves a compositional five-call advantage of 10.5 and 9.2 points over the inliner's flat serialization. It is positive in every seed, under exact matching, on average over audited rows, and with any skeleton or source left out; on the GSM8K test split the advantage is 4.1 points. Format comparisons therefore need per-format saturation checks before any representation-level interpretation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language model reasoning, program-based reasoning, output budgets, structured generation, evaluation methodology

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_eqNyOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies the effect output-token limits on programs generated to solve arithmetic word problems. It uses two JSON encodings: one specifies calls to functions from a predefined arithmetic library, while the other writes out those functions' operations. The authors call these “compositional plans” and “flat programs,” respectively. The authors use a interpreter to execute the generated programs to obtain numerical answers. The paper first compares the same flat-program checkpoints at different output limits. Allowing longer outputs substantially improves program validity, and truncating the longer outputs largely reproduces the short-budget failures. Then this paper provides analysis on different format, where the function-calling format retains an accuracy advantage when both formats receive enough tokens to finish.

Critical strengths
  1. Running the same checkpoints at different output limits shows that truncation explains much of the drop in program validity. The prefix check and the re-truncation of saved long outputs separate token-limit failures from errors in completed programs.

  2. The authors check confounds before reading higher validity as better reasoning: they break results down by program structure, fix incorrect training targets and labels, and rerun the format comparison.

Critical weaknesses
  1. the methodology is unclear and hard to understand.
  2. The basic setup is unclear. Terms such as “compositional plan,” “primitive,” and “subroutine” appear before any example of the model's output, and the only example is Figure 2 in the appendix. Table 1 and §4.4 answer different questions (output limits for the same checkpoints vs. separately trained models per format), and the “Skeleton,” “Augmented,” and “historical” labels are not explained.
  3. The contribution is narrow. That a token limit cuts off verbose programs is expected, and the accuracy gains are small: most five-call gains come from one structure across nine source problems, and GSM8K accuracy rises only from 26.9% to 28.4%. The format comparison is confounded by differences in quantity representation, and the paper never justifies its custom JSON language over executable Python.
Questions for authors
  1. Why use a custom JSON language instead of executable Python? Would the conclusions hold for Python programs that call predefined functions versus equivalent programs that inline them, with quantity representation held fixed?

  2. Beyond checking for truncation before interpreting validity, what broader conclusion about program-based reasoning do the results support, given that the five-call gains come mostly from S2 and the GSM8K gain is small?

Reviewer_4TB4OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper shows that a shared token cap can make longer output formats look like failed representations. Revisiting an earlier result where flat JSON programs seemed to fail on five-call math problems (which is unclear and not described in the paper), the authors show that raising the cap from 512 to 1536 tokens removes the truncation and recovers most of the apparent gap. They then conducted a set of surrounding analysis.

Critical strengths

The authors release their predictions and code, which helps reproducibility.

Critical weaknesses

Very Poor Presentation. The abstract reports a lot of numbers and relies on undefined terms (e.g., “skeleton-disjoint,” “exact-prefix”). “In a program-based mathematical reasoning benchmark whose…” is very obscure and we have no clue what this is referring to. It’s the same in Section 4.4, The first sentence, “The original 512-token study reported….” There is no citation to this sentence of the earlier study. Not to mention that the introduction only has 2 paragraphs without proper motivation but “This problem arises when comparing a flat inlined program…”

Limited novelty. The proposed contribution is diagnostic rather than methodological where it proposes no new method, model, training procedure, or format. Even the setup it diagnoses is insufficiently described The data in 3 setup describes: ‘The data start from 565 plans,…, proposed by Qwen3.5-27B and kept only if a deterministic executor reproduces the reference answer’ without proper description of the filtering process or any prompts.

Questions for authors

What exactly is the "In a program-based mathematical reasoning benchmark whose 512-token evaluation appeared to show flat JSON programs collapsing on five-call problems while compositional plans did not"?

Reviewer_nP3aOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper examines how a shared output-token cap affects comparisons between flat JSON programs and compositional arithmetic plans. Paired decoding of the same checkpoints, offline truncation, and forecasts at intermediate budgets identify truncation as the main source of the apparent five-call validity collapse. Structure-level analysis separates restored executability from correct computation. A semantics-preserving inliner exposes errors in historical training targets and enables a matched retraining comparison. With per-format saturation, the evaluated compositional system retains an accuracy advantage on the two benchmark sets and the full GSM8K test split. Additional experiments extend the truncation diagnosis to new computation structures and a second model family.

Critical strengths
  1. The truncation diagnosis is supported by a coherent sequence of interventions and predictions (Sections 4.1-4.3; Tables 1-4). The same-checkpoint comparisons isolate the output cap; offline truncation reproduces 98.6% of five-call correctness outcomes. Pairs in which the shorter output hits the token limit and exactly matches a prefix of the longer output account for 247 of 253 net correct gains. Forecasts made before intermediate-budget decoding agree with 97.9% of correctness outcomes. The structure-level results reveal that execution recovers broadly while most accuracy gains come from one skeleton. This establishes a useful distinction between incomplete outputs and completed programs that compute the wrong answer.

  2. The semantic controls materially improve the interpretation of format comparisons (Sections 4.4-6; Appendix G). Sun et al. (2025) study output-budget effects, and Fan (2026) examines prompted structured output. The added contribution here combines trajectory-level diagnosis in learned executable representations with target-equivalence and retraining controls. The corrected inliner accounts for casts, reassignment, and the final output reference; the audit shows that the historical converter changed the returned values of 286 of 4,795 training targets. Matched retraining and saturation checks then quantify the remaining accuracy difference between these two serialization systems. Data audits, source/skeleton-level uncertainty, and isolated repairs further support the interpretation.

Critical weaknesses
  1. The empirical support for the residual compositional advantage has limited structural breadth (Section 3; Appendix H, Tables 20–23). The two main five-call slices derive from the same nine source problems and six program skeletons. The full GSM8K evaluation also favors the compositional format, but its 206-question audited subset with new computation structures shows a difference of −1.1 percentage points (95% CI [−4.2, 1.9]). These results leave the benefit on unfamiliar program structures uncertain, limiting the scope of the representation comparison.
Questions for authors
  1. Please clarify the status and provenance of the historical comparison discussed in Sections 4.4 and 6: an earlier internal experiment, a released benchmark configuration, or a published result. Please identify its corresponding version and artifacts through the anonymous supplementary material. This would help assess the reach of the methodological correction and its relationship to existing evaluation practice.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.