acceptodds
Under review as a conference paper at ICLR 2027

SPIRAL: Learning to Search and Aggregate

Abstract

Language model reasoning can be substantially improved at test time via scaffolds that scale inference compute across different primitives—sequential reasoning within a trace, independently sampled parallel traces, and aggregation of multiple reasoning traces into a final response. During post-training, however, language models are optimized only for sequential reasoning within a single trace. We introduce Sequential-Parallel-Aggregative Reinforcement Learning (SPIRAL), a framework in which a language model is trained to use all three primitives end-to-end , as part of a unified inference compute pipeline. Concretely, the language model first samples a set of independent traces in parallel, each produced through sequential chain-of-thought reasoning, and then generates a final aggregation trace conditioned on those traces; all components are optimized end-to-end against the reward of the final aggregated response. To train this system, SPIRAL uses set reinforcement learning to teach models to produce a set of traces that are collectively useful for an aggregator and standard reinforcement learning to teach models to aggregate the set into improved final responses. Our experiments on reasoning tasks show that SPIRAL effectively scales with inference compute, outperforming GRPO by up to 11 scaling efficiency and 15% higher performance when all three compute primitives are scaled.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Inference Compute

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_K2GyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper uses only the reward of the final aggregated response, the candidate solutions (search traces) are trained with set RL and the synthesized solution (aggregation trace) is trained with standard RL. They are trained jointly. This makes test-time compute scaling where multiple solutions are sampled and then aggregated, show better performance than with GRPO. They experiment using Qwen3 4B model on mathematical reasoning benchmarks.

Critical strengths
  • The method is intuitive and simple. It could be adopted in other RL training pipelines.
Critical weaknesses
  • It is unclear where SPIRAL's gains come from. It misses an ablation on the set reward and the aggregation reward.
  • GRPO seems to scale no better than the base model, and the paper does not explain why.
Questions for authors
  • I would like to see an ablation on the set RL reward and the aggregation reward to understand where SPIRAL's gains come from.
  • It is unclear whether GRPO is expected to be this weak in this setting. Please report its improvement over the base model and clarify whether the GRPO configuration was sufficiently tuned.
  • The paper claims budget matching, but the chosen trace length (4096 vs. 8192) and number of rollouts (24 vs. 12) appear somewhat arbitrary. Why were these settings chosen, and how does GRPO perform at the same trace length/rollout numbers?
Reviewer_sjdtOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SPIRAL, an RL framework that trains a language model end-to-end to sample parallel reasoning traces and then aggregate them into a final answer, using only the reward of the aggregated output. Search traces are optimized with set RL via a marginal set advantage, and aggregation traces with standard GRPO-style RL; the authors also prove the set-RL gradient estimator is unbiased up to a scaling factor. On AIME-style math benchmarks with Qwen3-4B, SPIRAL outperforms compute-matched GRPO in pass@k scaling and under recursive self-aggregation.

Critical strengths
  1. The hypothesis could be right, as we re-use/aggregate some trajectories that originally be discarded, then the RL training efficiency could be improved. The Fig 6.a support this.
  2. Clean objective, and the gradient split actually falls out of the math. The objective in Eq. (3) only rewards the final aggregated answer. From there, Eq. (6) shows the gradient naturally breaks into two pieces: a set-RL term for the search traces and a standard RL term for the aggregation traces. So using different estimators for the two stages isn’t just bolted together; it comes straight from the objective
Critical weaknesses
  1. GRPO gets tested on something it was never trained to do. In Figs. 3b and 6, GRPO is run under recursive self-aggregation using the aggregation prompt (B.2), even though it never saw aggregation during training. So of course SPIRAL wins there. A fairer baseline would be GRPO that’s also trained to aggregate, or methods like Qi et al. (2025) or PaCoRe.
  2. There’s no ablation for the set-RL part, which is the main contribution. The obvious experiment is to swap the set-RL term (Algorithm 1, line 14) for plain per-trace GRPO advantages and keep the aggregation training the same. Without it, we can’t tell whether the gains come from set RL or just from training the model to aggregate.
  3. The core motivation is plausible but never tested. The paper argues that wrong traces can still contribute useful ideas once combined with others. That’s a reasonable idea, but prior aggregation work already shows aggregators can use wrong traces. What’s new here is the claim that set RL teaches the search policy to produce such helpful traces, and nothing in the paper checks that. A simple test would be aggregation accuracy on sets where every search trace is wrong, compared with a fresh attempt.
Questions for authors
  1. In recursive self-aggregation, how are the 8 traces at each level grouped into sets of 4 while keeping the population at 8? Are subsets sampled with replacement? And is the reported pass@1 averaged over all traces in the final population?
  2. How is the 36k minimum token budget for recursive self-aggregation in Fig. 6b computed? Also, what does the x-axis k represent for recursive self-aggregation in Fig. 6a?
  3. Could you compare against a baseline that is also trained to aggregate, such as GRPO with aggregation training, Qi et al. (2025), or PaCoRe? Currently GRPO is evaluated under recursive self-aggregation without ever having been trained on the aggregation prompt.
Reviewer_E465OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SPIRAL, an end-to-end reinforcement learning framework for training language models that jointly uses sequential reasoning, parallel search, and aggregation. The framework employs set-level RL to reward candidate traces based on their collective usefulness to a downstream aggregator, and standard RL to optimise the aggregation trace using the final answer's reward.

Critical strengths
  • The proposed method is interesting, which optimizes candidate sets for their collective utility rather than only individual trace correctness.
  • The authors train one model and report evaluation results on multiple benchmarks, showing the effectiveness of the proposed method.
Critical weaknesses
  • The experiments do not isolate the contribution of set-level credit assignment from aggregation training and increased diversity. It would be nice to see such analyses.
  • It would be interesting to see comparisons of training cost, which requires detailed accounting of token usage, rollout costs, and computational resources.
  • The current experiments focus on one model and mathematical reasoning, with limited evidence of generalisation.
Questions for authors
  1. How is the reported 11* scaling-efficiency improvement defined and calculated? I see a maximum number reported in Figure 3a (which lacks the main caption) is 10.6.
Reviewer_NLo9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes SPIRAL, an RL framework that trains a language model end-to-end over a test-time pipeline of three compute primitives: sequential reasoning within a trace, parallel sampling of independent "search traces", and an aggregation trace that synthesizes a set of candidates into a final answer. Search traces are optimized with set RL (credit assigned via marginal contribution to the aggregation reward, following Orney et al., 2026); the aggregation trace is optimized with standard RL using a per-set baseline; both are driven only by the final answer's reward. On Qwen3-4B-Instruct-2507 with math benchmarks, SPIRAL reportedly scales much better with test-time compute than token-matched GRPO — up to 11x pass@k scaling efficiency, and up to 15% higher accuracy when parallel and aggregative compute are scaled jointly (though see W5 on this number).

Critical strengths
  • The problem is real and well framed: reasoning RL trains single sequential traces, while practical deployment wraps the model in sampling-plus-aggregation pipelines. Learning the pipeline end-to-end instead of hand-designing it is a direction worth pursuing.
  • The gradient decomposition into a set-RL term and a standard-RL term is clean, and the implementation choices (per-set baseline for aggregation, marginal set advantage for search traces) are sensible and clearly described.
  • The GRPO baseline is matched in sampled tokens, and the improvement shows up consistently in three different evaluations (pass@k, recursive self-aggregation, token-budget curves), so the main effect is unlikely to be an artifact of one particular metric.
  • The entropy (Fig. 4) and diversity (Fig. 5) analyses at least attempt to explain why the method helps, rather than only reporting final numbers.
  • Prop. 3.1 attempts to establish the unbiasedness of the set-RL gradient estimator (up to a learning-rate rescaling) — more theoretical care than typical for this kind of paper, even though the guarantee does not actually cover the implemented estimator (W4).
Critical weaknesses

W1. One model, one scale, one domain. All results are on Qwen3-4B-Instruct-2507, LoRA rank 32, math only. The two questions that matter most for judging generality are not addressed: does the gap shrink at larger scale (8B/14B), where the base model already aggregates reasonably well? Does it hold for other model families? The limitations section discusses verifiable rewards but says nothing about scale or model family.

W2. No ablation separating the two training components. The paper's central claim is that jointly training search and aggregation is what matters, but there is no control that trains only the aggregator over an untrained search policy (a PaCoRe-style variant — the cited closest work). Without it, we cannot tell how much of the gain comes from set-RL training of search traces versus from training aggregation alone. There is also no sensitivity analysis for N1=8, n=4, K=4, N2=4, even though these directly determine sample complexity and the variance of the marginal advantages.

W3. "Compute matched" is only token-matched. Each aggregation trace is generated over a ~16k-token concatenated context, so its per-token cost (prefill + attention) is far higher than a GRPO generation token. Wall-clock, memory, and FLOPs are never reported, and the intro's claim that sample complexity is "comparable to standard RL" understates the real training cost. Separately, GRPO traces are capped at 8192 tokens while SPIRAL search traces get 4096 — a per-trace sequential depth difference that is a confound in itself and is not discussed.

W4. Prop. 3.1 does not cover the implemented estimator. The proof assumes K sets sampled from ordered n-tuples; the implementation samples unordered sets, implicitly assuming f_spiral is symmetric — an assumption the authors themselves note fails due to ordering bias (App. D). Randomizing candidate order in the aggregation prompt is a cheap fix; at minimum the gap should be stated in the main text rather than left in the appendix.

W5. Statistics. AIME 2025/2026 have 30 problems each; no error bars, no seeds, and the headline numbers are "up to" point-wise maxima. The abstract's "15% higher" does not match the 13.5% reported in Sec. 4 and annotated in Fig. 6b.

Minor: (a) the length-penalty advantage hack (Eq. 9) is not ablated, and it is not stated whether GRPO gets the same treatment; (b) it is unclear whether the Fig. 6b token budget counts the input tokens the aggregator consumes when reading candidates — if not, aggregation-based methods get a systematically flattering cost curve; (c) the aggregation behavior is driven by a heavily engineered prompt (B.2), which sits awkwardly with the "no hand-designed scaffold" framing; (d) Fig. 5 is judge-dependent, and the base model in fact shows more strategy clusters than SPIRAL under all three judges, so the diversity claim should be stated more cautiously (pass@k remains the stronger evidence); (e) typos: duplicated "compute pipeline" in the Fig. 2 caption, "higher notably higher" in Sec. 4, and the limitations paragraph jumps from "First" to "Third".

If the authors can provide the aggregation-only ablation (W2) and honest cost accounting (W3), I would be willing to raise my score.

Questions for authors

Q1. What are the actual wall-clock training time and total FLOPs of SPIRAL relative to GRPO?

Q2. Are both methods trained with LoRA rank 32 via Tinker? Does full-parameter fine-tuning change the conclusions?

Q3. With only the aggregation stage trained (search policy frozen), how much of the gain remains?

Q4. How do the results change at 8B/14B scale? Any sign of diminishing returns with a stronger base model?

Q5. Which is the correct headline aggregation number — the abstract's 15% or Sec. 4's 13.5% — and under what exact setting was each measured?

Q6. How sensitive is performance to K and n, given their effect on the marginal-advantage variance?

1 comment

I appreciate the reply from the author; I will maintain the current score.

Comments on the paper

1 comment

The author-reviewer discussion period will end soon. Please review and acknowledge the authors’ responses. If your concerns have been addressed, please reconsider your score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.