- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies the effect output-token limits on programs generated to solve arithmetic word problems. It uses two JSON encodings: one specifies calls to functions from a predefined arithmetic library, while the other writes out those functions' operations. The authors call these “compositional plans” and “flat programs,” respectively. The authors use a interpreter to execute the generated programs to obtain numerical answers. The paper first compares the same flat-program checkpoints at different output limits. Allowing longer outputs substantially improves program validity, and truncating the longer outputs largely reproduces the short-budget failures. Then this paper provides analysis on different format, where the function-calling format retains an accuracy advantage when both formats receive enough tokens to finish.
Critical strengths
-
Running the same checkpoints at different output limits shows that truncation explains much of the drop in program validity. The prefix check and the re-truncation of saved long outputs separate token-limit failures from errors in completed programs.
-
The authors check confounds before reading higher validity as better reasoning: they break results down by program structure, fix incorrect training targets and labels, and rerun the format comparison.
Critical weaknesses
- the methodology is unclear and hard to understand.
- The basic setup is unclear. Terms such as “compositional plan,” “primitive,” and “subroutine” appear before any example of the model's output, and the only example is Figure 2 in the appendix. Table 1 and §4.4 answer different questions (output limits for the same checkpoints vs. separately trained models per format), and the “Skeleton,” “Augmented,” and “historical” labels are not explained.
- The contribution is narrow. That a token limit cuts off verbose programs is expected, and the accuracy gains are small: most five-call gains come from one structure across nine source problems, and GSM8K accuracy rises only from 26.9% to 28.4%. The format comparison is confounded by differences in quantity representation, and the paper never justifies its custom JSON language over executable Python.
Questions for authors
-
Why use a custom JSON language instead of executable Python? Would the conclusions hold for Python programs that call predefined functions versus equivalent programs that inline them, with quantity representation held fixed?
-
Beyond checking for truncation before interpreting validity, what broader conclusion about program-based reasoning do the results support, given that the five-call gains come mostly from S2 and the GSM8K gain is small?