- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a frozen adviser that observes a student's latest reasoning paragraph and provides optional structural guidance, such as checking a claim, continuing an approach, switching strategies, or finishing.
Critical strengths
The distinction between non-binding advice and externally controlled reasoning is interesting. The experiments cover nine benchmarks, and the authors include meaningful controls and paired trajectory analyses. The 1.5B greedy-decoding accuracy improvement is substantial, particularly on MATH500 and GPQA-Diamond.
Critical weaknesses
The paper suffers from the following issues:
(1) The paper claims a 45.2% reduction in generated tokens for the 1.5B student. However, the metric excludes adviser-generated tokens, adviser prefill, and the overhead of repeatedly restarting student decoding. So, for a fair comparison, you need to include these missed generated tokens. This could be concerning for the 7B model where the token reduction is 21.1%.
(2) The strongest results come from 1.5B greedy decoding. However, the accuracy improvement decreases to approximately 1 percentage point for the 7B model and 1.4 points under sampled decoding for 1.5B model. The 7B results are also inconsistent over various datasets. These findings raise concerns about the privilege of the proposed method over the SOTA methods. In Table 3, page 7, the paper compares Cornerman with ACTS, DEER, JET, and LC-R1. However, its advantage over the strongest alternatives is relatively small. ACTS achieves nearly the same accuracy with approximately 46% fewer student-generated tokens.
Questions for authors
Please check the Critical Weaknesses section.
2 comments
While the end-to-end measurements and budget-controlled comparisons address some of my concerns, they also reveal substantially higher computational costs, particularly for the 7B model. Although the 1.5B latency and MATH500 results are encouraging, the limited accuracy gains and inconsistent efficiency improvements across settings leave my overall concerns only partially resolved. I therefore maintain my rating.
Response to Reviewer dnRc
Thank you for recognizing the distinction between non-binding advice and external control. We address the two concerns with full-system measurements and additional budget-controlled comparisons.
1. Complete inference cost
The reported 45.2% and 21.1% reductions measure student-generated output across nine benchmarks. We have now measured end-to-end cost on MATH500, including student and adviser prefill and decoding, and student-context recomputation after advice replacement.
Measurements use one A800-80GB with vLLM 0.7.3 and prefix caching. Batch time covers all 500 problems; single-query latency averages the same 50 questions at concurrency 1. Estimated FLOPs sum 2 × model parameters × uncached processed tokens over both models, excluding attention operations.
Adviser generation averages 387 and 338 tokens per problem for the 1.5B and 7B students, respectively; these are included in the accounting above. After including adviser computation and student-resumption overhead, 1.5B Cornerman completes all 500 problems in 26.2 rather than 72.3 minutes, a 2.8× speedup. Mean single-query latency falls from 67.8 to 33.1 seconds, a 51.2% reduction, with 1.32× the estimated FLOPs. For 7B, single-query latency falls from 48.2 to 38.0 seconds (21.2%), while batch runtime and estimated FLOPs increase. The full-system measurements therefore show speedups in both batch processing and single-query response at 1.5B; the 7B runtime benefit is confined to single-query response.
2. Gains across settings and comparison with alternatives
7B and sampled decoding. The gains are smaller than for 1.5B greedy decoding, but accuracy improves alongside shorter output in both settings. With the same frozen 1.5B adviser, the 7B student gains approximately 1 percentage point in macro accuracy while generating 21.1% fewer student tokens. The effect is benchmark-dependent, with GPQA-Diamond improving from 40.4% to 46.5%.
Under sampled decoding, Cornerman raises macro accuracy from 48.4% to 49.8% while reducing mean student output from 9,115 to 6,383 tokens (30.0%). The smaller accuracy gain is therefore accompanied by substantially shorter reasoning, with the student frozen and no new solution steps supplied by the adviser.
ACTS, JET, and s1. ACTS does produce approximately 46% fewer student tokens in Table 3. This comparison excludes its controller computation, just as it excludes our adviser's computation. We therefore compared costs including both models on MATH500 and evaluated Cornerman with 8k and 16k student budgets. We also applied s1-style budget forcing (Muennighoff et al., 2025) to the same Full-CoT student:
The paper's standard 32k Cornerman result remains 84.4%; the 8k/16k Cornerman rows are additional experiments. JET's row reports a new evaluation of its released checkpoint with the same final-answer scorer. ACTS costs include its 4B controller; its runtime was not measured on this card. The other runtimes use the hardware above.
At 16k, Cornerman exceeds ACTS by 3.2 percentage points at comparable estimated compute (34.8 versus 36.3 TFLOPs/problem). JET uses fewer estimated FLOPs, while Cornerman at 8k achieves higher accuracy and shorter batch runtime. At the same 8k student budget, Cornerman also improves on s1-style budget forcing by 1.8 points with similar batch runtime (9.9 versus 10.0 minutes), though higher estimated FLOPs. Unlike JET's reasoner post-training, Cornerman trains a separate adviser shared across frozen students.
Additional task coverage. On 1,000 MMLU-Pro test questions sampled across all 14 subjects, the same adviser improves the 1.5B Full-CoT student's accuracy from 22.5% to 27.2%, while reducing mean student output from 18,221 to 5,649 tokens (69.0%). Accuracy improves in 12 of the 14 subjects, including business, law, and psychology.
Together, these results support the value of a shared local adviser for frozen students: gains persist under sampling and at 7B, and the budget-controlled evaluation shows higher MATH500 accuracy than ACTS at comparable estimated compute.