acceptodds
Under review as a conference paper at ICLR 2027

Learning Prompt Chains for Frozen LLMs: Inter-Call Orchestration Beyond Single-Call Reasoning

Abstract

Reinforcement learning has made Chain-of-Thought prompting an effective form of test-time computation, but it remains unclear whether multi-call prompting strategies can be learned in the same way. We study prompt chaining as a trainable orchestration policy for frozen large language models (LLMs). We introduce Chainer, a lightweight model that maps each user request to an open-loop sequence of self-contained sub-prompts executed by a frozen LLM. Chainer is first initialized by supervised fine-tuning and then optimized with GRPO using end-task rewards, chain-quality signals, validity bonuses, and a dynamic step cost. Across six benchmarks, learned chaining improves a non-reasoning executor on four tasks. With reasoning enabled, it remains complementary on five, improving Arena-Hard (+17.3), AIME (+10.0), LiveCodeBench (+8.2), GPQA (+2.0), and AA-Omniscience (-2.9; lower is better), while degrading IFBench. Experiments shows that these gains are not redundant with longer single-call reasoning: gains persist when token budgets are matched. Since executor weights are fixed, the improvements arise solely from optimizing the prompt trajectory. These results position learned prompt chaining as a distinct axis for scaling frozen LLM systems.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Artificial Intelligence, Reasoning, Reinforcement Learning, Test Time Compute

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ndVsOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a learnable prompt-chaining method for frozen LLMs. A Chainer is trained with SFT and GRPO to generate an input-dependent, variable-length prompt sequence, which a frozen executor follows through multiple calls. Unlike dynamic agents, the Chainer produces the full chain before execution, making the method an open-loop orchestration approach. Across six benchmarks, chaining improves point-estimate performance on four tasks without reasoning and five with reasoning, while notably degrading IFBench. The paper also studies chain length, cross-model transfer, and inference cost.

Critical strengths
  1. The paper studies a timely and practically relevant problem. It investigates how to improve frozen LLMs through learned, input-conditioned orchestration, offering a complementary direction to scaling single-call reasoning.

  2. The evaluation is broad and includes meaningful analyses. Experiments span six benchmarks and examine reasoning-enabled and disabled settings, cross-executor transfer, chain behavior, and inference cost.

  3. The paper reports limitations and failure modes transparently. It highlights instruction-following degradation, increased inference cost, heterogeneous transfer, and safety risks, giving a more balanced picture of when prompt chaining may or may not help.

Critical weaknesses
  1. The source of the gains is not sufficiently isolated. The current ablations do not distinguish learned decomposition from prompt rewriting or repetition, accumulated context, and additional executor calls. Comparisons with equally budgeted repeated prompts, fixed chains, and rewriting without intermediate execution would clarify what the Chainer contributes.

  2. The cost comparison does not match end-to-end compute. Chaining adds planner generation, executor input tokens, and latency, but the analysis primarily matches thinking-token budgets or executor completion tokens. The paper therefore does not establish whether chaining outperforms reasoning-only at equal total cost.

  3. Some empirical claims are stronger than the statistical evidence. The confidence intervals for AIME and GPQA include zero in both reasoning settings, and AIME has only 30 examples. The paper should distinguish positive point estimates from statistically supported gains, address multiple comparisons, and qualify claims where uncertainty is not reported, including AA-Omniscience.

  4. The contribution of the LLM-judged chain reward is unclear. The reward may favor chains that appear effective to the judge without improving task performance. Ablations removing or varying this reward, along with independent evaluation of open-ended results, are needed to establish that gains reflect better orchestration rather than judge-specific optimization.

Questions for authors

See weaknesses.

Reviewer_VynJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies learned prompt chaining as an inter-call orchestration test-time compute strategy for frozen LLMs, which is different from single-call intra-call reasoning. The authors propose Chainer, a lightweight LoRA-tuned model (Qwen-3 4B) to map a user query into an open-loop sequence of self-contained sub-prompts executed sequentially by a frozen LLM. The model is initialized via SFT and optimized with GRPO using outcome rewards, chain-quality judge signals, and a dynamic step penalty. Chainer improves a frozen executor on 4 of 6 benchmarks without reasoning and 5 of 6 with intrinsic reasoning enabled (though degrading IFBench). Control ablations confirm that these gains persist at matched thinking-token budgets, demonstrating that multi-call orchestration is complementary to longer single-call reasoning.

Critical strengths
  1. The conceptual framing of inter-call and intra-call test-time compute is interesting.
  2. The RL reward is well-formulated from multiple angles.
  3. The SFT-vs-GRPO ablation is informative.
Critical weaknesses
  1. The Novelty of the Paper is Insufficient The idea of the paper is not new. Prompt refinement has been widely explored in the literature, and using GRPO for model training is also not new.

  2. Inconsistent Numbers and Misplaced Bold Formatting in Table 1 (Main Results) There is a notable discrepancy in the headline numbers for Arena-Hard v2 (Reasoning OFF). In Table 1, Baseline is listed as 37.4 and Chained is 32.2. However, in Table 3, Table 4, Table 7, and Table 12, the exact same comparison is reported with Baseline = 32.2 and Chained = 37.4. It appears the two columns were accidentally swapped in Table 1 for Arena-Hard.

  3. Prohibitive Training Cost vs. Poor Executor Transferability The paper motivates Chainer as a lightweight way to improve frozen LLMs, potentially including larger or externally hosted models. However, Appendix A.13 reveals that training the Chainer for a single 4B executor took 10 wall-clock days on 3 NVIDIA B200 GPUs (~30 B200 GPU-days) across GRPO I and II. Because GRPO requires sampling 10 multi-step walks per prompt and running sequential rollouts through the executor plus a 30B LLM judge at every training step, the training pipeline is computationally massive.

Questions for authors

N/A

Reviewer_TVjcOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Chainer, a lightweight model that learns to split user requests into multi-step prompt sequences for a frozen LLM. Trained via supervised fine-tuning and reinforcement learning, it improves performance on 5/6 benchmarks — notably Arena-Hard (+17.3%) and AIME (+10.0%) — showing that learned inter-call orchestration complements single-call reasoning without modifying model weights.

Critical strengths
  • Rigorous disentanglement of inter-call vs. intra-call compute. The token-matched analysis (Figure 2) shows chaining outperforms reasoning-only at every budget level, directly ruling out the "just more tokens" explanation. This is the paper's most consequential evidence — it establishes learned prompt chaining as a structurally distinct scaling axis rather than a proxy for longer generation.
  • Comprehensive and honest evaluation. Six diverse benchmarks, four execution regimes, and 11 strong baselines (including RSA, ToT, Self-Refine) provide broad coverage. Critically, the paper transparently reports failures — degradation on IFBench, hallucination sensitivity, and a dramatic jailbreak increase (27.5% → 76.1%): lending credibility to the positive results and enabling informed assessment of the method's scope.
Critical weaknesses
  • As the performance scales with test time compute, one should control the compute budget/usage for comparing the proposed approach with CoT baseline.
  • The proposed method show mixed results with CoT baseline.
  • The model used in the experiments are small, it is unclear whether the pattern holds on frontier models. If the intention is for SLM research, one should make it clear and limit the scope.
Questions for authors
  • What model do you use in Section 5.1? You mention 'frontier' in the plot caption, is it a correct statement?
  • Why do you pick the Arena-Hard v2 in Section 5.1? The benchmark favors the proposed approach so I find it unconvincing.

Comments on the paper

1 comment
Area_Chair_KQbCOpenReview ↗

Reminder: Reviewer Engagement During the Discussion Period

Dear Reviewers,

As the discussion period is drawing to a close, could you please review the authors' rebuttal and provide your feedback at your earliest convenience?

Active engagement in discussions with the authors is an essential part of the peer-review process and plays an important role in ensuring the success of the conference. Your participation would be greatly appreciated.

Thank you for your time and contributions.

Best regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.