- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a learnable prompt-chaining method for frozen LLMs. A Chainer is trained with SFT and GRPO to generate an input-dependent, variable-length prompt sequence, which a frozen executor follows through multiple calls. Unlike dynamic agents, the Chainer produces the full chain before execution, making the method an open-loop orchestration approach. Across six benchmarks, chaining improves point-estimate performance on four tasks without reasoning and five with reasoning, while notably degrading IFBench. The paper also studies chain length, cross-model transfer, and inference cost.
Critical strengths
-
The paper studies a timely and practically relevant problem. It investigates how to improve frozen LLMs through learned, input-conditioned orchestration, offering a complementary direction to scaling single-call reasoning.
-
The evaluation is broad and includes meaningful analyses. Experiments span six benchmarks and examine reasoning-enabled and disabled settings, cross-executor transfer, chain behavior, and inference cost.
-
The paper reports limitations and failure modes transparently. It highlights instruction-following degradation, increased inference cost, heterogeneous transfer, and safety risks, giving a more balanced picture of when prompt chaining may or may not help.
Critical weaknesses
-
The source of the gains is not sufficiently isolated. The current ablations do not distinguish learned decomposition from prompt rewriting or repetition, accumulated context, and additional executor calls. Comparisons with equally budgeted repeated prompts, fixed chains, and rewriting without intermediate execution would clarify what the Chainer contributes.
-
The cost comparison does not match end-to-end compute. Chaining adds planner generation, executor input tokens, and latency, but the analysis primarily matches thinking-token budgets or executor completion tokens. The paper therefore does not establish whether chaining outperforms reasoning-only at equal total cost.
-
Some empirical claims are stronger than the statistical evidence. The confidence intervals for AIME and GPQA include zero in both reasoning settings, and AIME has only 30 examples. The paper should distinguish positive point estimates from statistically supported gains, address multiple comparisons, and qualify claims where uncertainty is not reported, including AA-Omniscience.
-
The contribution of the LLM-judged chain reward is unclear. The reward may favor chains that appear effective to the judge without improving task performance. Ablations removing or varying this reward, along with independent evaluation of open-ended results, are needed to establish that gains reflect better orchestration rather than judge-specific optimization.
Questions for authors
See weaknesses.
Reminder: Reviewer Engagement During the Discussion Period
Dear Reviewers,
As the discussion period is drawing to a close, could you please review the authors' rebuttal and provide your feedback at your earliest convenience?
Active engagement in discussions with the authors is an essential part of the peer-review process and plays an important role in ensuring the success of the conference. Your participation would be greatly appreciated.
Thank you for your time and contributions.
Best regards,
AC