- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies the trade-off of accuracy and reasoning length from several reasoning-compression methods. The authors show that the average response length and accuracy of each trained model can be fit by a low-dimensional saturating curve. They also further report that correctness matrices across problems and compressed models are dominated by a shared direction that transfers to held-out problems, and that this direction varies monotonically but nonlinearly with response length.
Critical strengths
- conduct a broad and careful comparison across methods, models and task types.
- The low-dimensional regularities are presented clearly and is useful.
Critical weaknesses
- The topic itself is very old, has been explored much, and this paper doesn't raise something much insightful.
- The main regularity is not surprising relative to the experimental construction. If generating each model family just by sweeping one scalar length, it will do have a dominant one-dimensional direction.
- Do not have much practical use. Need 3-9 training point according to the paper to rebuild the curve.
Questions for authors
- When two or more interventioon axes are mixed, still mainly focus on one direction?
- How does the proposed fit compare with linear interpolation, isotonic regression, and standard saturating-curve baselines?
- Does the different setting of max length of training and eval matters? Why the code task (LiveCodeBench) show weaker structure?
- What decision (controller choice, choice of setting) would a practitioner make differently because of these findings? Can you demonstrate one?
3 comments
Thanks for your reply. Yeah I think you solve part of my problems, proving that the pattern exists in a wide field and perhaps are not restricted to training settings. But I still didn't see the innovation here as it is common sense that there exists best length for each problem and certainly moderate compression may remove redundant or error-prone reasoning and improve accuracy, whereas excessive compression may eventually hurt it, and I didn't know what you want to tell. In LEASH or LASER, they do such kind of training because they realize this kind of problem and thus find a way to get the optimized length for each problem, so if you base on these and then do different length, I do not see what else is revealed. Actually I share same concern with other reviewers, I do not believe the remaining contribution currently meets the ICLR acceptance bar so I will keep my score.
Thanks for spelling this out. We agree that the basic intuition is familiar: moderate compression can help, while excessive compression can hurt. What we want to add is a way to measure this trade-off and understand where it comes from at the question level.
First, knowing that an optimal length exists does not tell us how much accuracy we pay for the next token saved, or which method is better when they end up at different lengths. LEASH still leaves target-length selection to the user (§6 of the LEASH paper). Our fit gives a differentiable curve and predicts held-out points more accurately than common interpolation methods. For DSDT-1.5B with LP, median RMSE across the six math datasets is 1.66 percentage points, versus 2.37 for linear interpolation and 2.46 for PCHIP. This lets us estimate where further compression becomes costly and compare methods at the same actual length (§4.3).
Second, our question-level results offer another way to think about adaptive compression. The approximate shared shape helps explain why the average curve stays simple, even though questions have different accuracy levels and response amplitudes. LASER-D groups questions by current accuracy and gives shorter targets to easier groups. Current accuracy describes performance at one setting. On the other hand, with the shared shape and its scale fixed, our amplitude describes how question ’s accuracy changes across trained compression policies. It is therefore closer to the compression response we want to measure. It provides a different perspective: considering sensitivity to compression alongside current difficulty.
We have incorporated the corresponding clarifications and analyses from our rebuttal into the revised PDF. The updated passages and added results are marked in blue.
We would be grateful for a brief follow-up on whether these updates resolve your concerns or whether further clarification is needed.