- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
AIPO trains a reasoning policy to request verification, knowledge, or subproblem-solving assistance during its own rollouts. It learns from both student and collaborator tokens, using a one-hot surrogate and a gradient-preserving lower bound for external-token updates. The resulting policy is evaluated without collaborators. Experiments cover eight tasks, self and stronger collaborators, expert-guided baselines, component ablations, a 32B extension, and training-cost comparisons.
Critical strengths
-
A novel LLM RL algorithm of AIPO is proposed via collecting verification, knowledge, or subproblem-solving assistance during its own rollouts.
-
The empirical comparison is reasonably broad and includes LUFFY and OPSD, rather than only ordinary RL. Improvements across several task families and collaborator settings make the result more persuasive than an isolated mathematics gain.
-
The final evaluation removes collaborators, directly testing the intended transfer to independent reasoning. The external-token ablations and the decline in low-probability external tokens provide useful evidence about the proposed learning mechanism, beyond showing that assisted rollouts are easier.
Critical weaknesses
-
The rationale for the specific interaction design of AIPO is not sufficiently developed. While the motivation for requesting help during reasoning is understandable, it is less clear why these three forms of assistance are the appropriate choice and whether actively choosing when to request them is necessary. The existing ablations show that the components help, but do not clearly separate the benefit of active interaction from simply receiving additional guidance.
-
The student models are relatively dated. Qwen3 is used as a collaborator rather than as the student, while the 32B extension still uses Qwen2.5. Experiments with a more recent student would help establish whether the gains persist when the policy already has stronger reasoning capabilities and may benefit less from external assistance.
-
The larger-model experiments mainly compare against SFT and GRPO. Although LUFFY and OPSD are included in the main experiments. More recent and competitive LLM RL methods should be included in the comparison.
-
LLM RL training is already resource-intensive, and AIPO additionally requires serving collaborator models and processing longer interaction trajectories. This adds memory and compute demands, particularly when using a larger collaborator.
Questions for authors
Please see Critical Weaknesses.
8 comments
Response to Reviewer a8W2 (Part 2 / 3)
W2: The student models are relatively dated
We added experiments with more recent LLMs: Qwen3-8B in non-thinking mode (Qwen3-8B-NoThinking) as the policy model. The results are shown in the table below and added in Appendix C.7. Within each collaborator setting, AIPO outperforms all baselines on every benchmark.
Averaged over the eight benchmarks, AIPO with a same-scale Qwen3-8B-NoThinking collaborator improves the original model from 40.3 to 46.6. This is 1.3 points above the strongest baseline that does not use a larger collaborator, Dr.GRPO (45.3), and 1.9 points above LUFFY with the same collaborator (44.7). With the larger Qwen3-30B-A3B-Instruct-2507 collaborator, AIPO reaches 47.9, 1.3 points above LUFFY (46.6). AIPO with the same-scale collaborator already matches LUFFY with the larger collaborator (46.6 vs. 46.6). These results suggest that a policy with stronger reasoning ability can still benefit from external assistance.
| Methods | AIME24 (Avg@16) | AIME25 (Avg@16) | MATH500 (Avg@4) | LMB (Avg@4) | GPQA-D (Avg@4) | MBPP (Avg@4) | LCB (Avg@4) | RG (Avg@4) |
|---|---|---|---|---|---|---|---|---|
| Qwen3-8B-NoThinking | ||||||||
| Original | 29.1±0.1 | 20.9±0.3 | 87.4±0.2 | 22.4±0.4 | 44.4±0.1 | 72.8±0.3 | 28.7±0.1 | 16.4±0.2 |
| GRPO | 39.7±0.4 | 30.6±0.3 | 89.9±0.1 | 25.1±0.2 | 48.6±0.5 | 74.5±0.1 | 30.7±0.4 | 20.9±0.3 |
| PRIME | 38.8±0.6 | 30.1±0.4 | 89.0±0.3 | 23.0±0.5 | 46.5±0.7 | 72.1±0.4 | 29.2±0.6 | 18.8±0.3 |
| Dr.GRPO | 40.0±0.3 | 30.9±0.2 | 90.1±0.1 | 25.2±0.1 | 48.8±0.4 | 74.7±0.1 | 31.0±0.3 | 21.4±0.1 |
| Qwen3-8B-NoThinking Qwen3-8B-NoThinking | ||||||||
| SFT | 37.1±0.5 | 29.1±0.4 | 87.6±0.3 | 22.7±0.5 | 44.7±0.7 | 73.0±0.4 | 28.9±0.6 | 16.6±0.3 |
| OPSD | 39.3±0.4 | 30.5±0.3 | 89.3±0.1 | 24.1±0.3 | 48.7±0.5 | 73.7±0.2 | 29.8±0.4 | 19.0±0.3 |
| LUFFY | 40.1±0.6 | 30.2±0.5 | 89.2±0.3 | 24.5±0.4 | 49.2±0.6 | 74.1±0.3 | 30.1±0.5 | 20.4±0.4 |
| AIPO | 42.2±0.2 | 32.7±0.1 | 90.9±0.1 | 26.0±0.1 | 51.4±0.3 | 75.4±0.1 | 31.7±0.2 | 22.3±0.1 |
| Qwen3-8B-NoThinking Qwen3-30B-A3B-Instruct-2507 | ||||||||
| SFT | 39.0±0.5 | 30.6±0.4 | 88.8±0.3 | 23.9±0.5 | 45.2±0.7 | 73.3±0.4 | 29.3±0.6 | 17.2±0.3 |
| OPSD | 41.7±0.4 | 32.0±0.3 | 90.3±0.1 | 25.4±0.3 | 50.3±0.5 | 74.5±0.2 | 30.7±0.4 | 21.1±0.3 |
| LUFFY | 42.4±0.6 | 32.6±0.4 | 91.1±0.2 | 26.1±0.4 | 51.5±0.4 | 75.4±0.3 | 31.7±0.5 | 22.0±0.4 |
| AIPO | 44.0±0.6 | 33.6±0.4 | 91.7±0.2 | 28.2±0.4 | 52.4±0.4 | 76.2±0.3 | 33.3±0.5 | 23.9±0.4 |
W3: The 32B experiments should include more competitive RL baselines
Thank you for the suggestion. We added LUFFY and OPSD to the Qwen2.5-32B-Instruct experiments, using the same Qwen2.5-32B-Instruct collaborator. The results are shown in the table below where AIPO outperforms all baselines on every benchmark.
Averaged over the eight benchmarks, AIPO reaches 57.8, compared with 55.6 for LUFFY, 53.5 for GRPO, 52.2 for OPSD, and 50.7 for SFT. This is 2.2 points above the strongest baseline (LUFFY) and 9.7 points above the original model (48.1), indicating that the advantage of AIPO persists at the 32B scale.
| Methods | AIME24 (Avg@16) | AIME25 (Avg@16) | MATH500 (Avg@4) | LMB (Avg@4) | GPQA-D (Avg@4) | MBPP (Avg@4) | LCB (Avg@4) | RG (Avg@4) |
|---|---|---|---|---|---|---|---|---|
| Qwen2.5-32B-Instruct Qwen2.5-32B-Instruct | ||||||||
| Original | 29.9±0.4 | 28.6±0.4 | 86.4±0.3 | 27.8±0.5 | 62.6±0.5 | 80.6±0.5 | 40.2±0.4 | 28.6±0.4 |
| SFT | 32.6±0.4 | 31.3±0.4 | 88.3±0.2 | 30.3±0.4 | 65.3±0.5 | 83.6±0.4 | 42.9±0.4 | 31.1±0.4 |
| OPSD | 34.3±0.5 | 33.1±0.5 | 88.4±0.2 | 32.0±0.5 | 67.0±0.4 | 85.3±0.5 | 44.6±0.4 | 32.8±0.5 |
| GRPO | 35.7±0.5 | 34.5±0.5 | 88.6±0.2 | 33.5±0.5 | 68.4±0.4 | 86.7±0.5 | 46.0±0.4 | 34.2±0.5 |
| LUFFY | 37.4±0.6 | 37.1±0.5 | 88.9±0.2 | 36.2±0.5 | 70.5±0.5 | 88.5±0.4 | 47.2±0.4 | 39.1±0.5 |
| AIPO | 39.1±0.4 | 39.7±0.4 | 89.3±0.2 | 38.9±0.4 | 72.6±0.4 | 90.2±0.3 | 48.4±0.4 | 44.0±0.4 |
Response to Reviewer a8W2 (Part 3 / 3)
W4: Serving collaborators adds memory and compute, particularly with a larger collaborator
We agree that serving and querying collaborators adds training cost. But serving collaborators adds a moderate, training-only overhead. AIPO costs 11% more GPU hours than GRPO with a same-scale collaborator and about the same as teacher-based baselines with a larger one, outperforms GRPO under a matched compute budget, and needs no collaborator at inference.
First, the overhead is moderate with a same-scale collaborator, and with a larger collaborator it is on par with teacher-based baselines that use the same model. With a same-scale Qwen2.5-7B-Instruct collaborator, AIPO adds 11% total GPU hours (149 → 166) and 7% peak memory (286 → 307 GB) over GRPO, while raising the average from 33.3 to 35.4 (Table 9). This matches LUFFY with the larger Qwen3-30B-A3B teacher (35.4) at about half its GPU hours (166 vs. 310). With Qwen3-30B-A3B as the collaborator, the total cost of AIPO is comparable to OPSD (329 vs. 326 GPU hours) and LUFFY (329 vs. 310, including offline trajectory generation), while its average is 2.8 and 1.7 points higher, respectively.
Second, the gains persist under matched compute. With an identical GPU-hour budget, AIPO outperforms GRPO in all four domains (Table 10).
Third, the overhead is adaptive and limited to training. The policy decides when to query collaborators, and per-batch calls decrease by about 30% from step 50 to step 200 (Figure 6). At inference, the policy reasons alone, with no additional memory, latency, or API cost.
We hope our responses could address your concerns. If you have any further questions, please do not hesitate to let us know.
Response to Reviewer a8W2 (Part 1 / 3)
We thank the reviewer for the constructive review and for recognizing the novelty of AIPO, the extensive comparisons, and the collaborator-free evaluation. We address each weakness below.
W1 (part 1): Why these three forms of assistance are the appropriate choice
The three collaborators provide complementary support, but AIPO does not depend on them: a single general-purpose collaborator achieves the same average performance (35.4 over the eight benchmarks, please see our response to Reviewer QhTy, W1).
Why three forms of assistance. 1) It provides more stable and structured supervision during training. AIPO enables the student model to improve through active interaction with external agents. The key question is therefore how to design interactions that effectively facilitate learning. Inspired by the key capabilities required in complex reasoning, we explicitly introduce three specialized agents targeting knowledge acquisition, verification, and reasoning. By decomposing the interaction into these complementary capabilities, AIPO provides clearer and more structured learning signals, reducing the difficulty of deciding what kind of assistance is needed. This mainly improves the training process, making learning more stable and interaction-efficient without being essential to the final capability of the student model. The training dynamics below support this effect (detailed figures are added in Appendix C.6). Around step 100, AIPO reaches 0.453 batch task accuracy, compared with 0.260 for the single-collaborator variant, while using 25.2% fewer interactions per batch. Around step 200, both reach comparable accuracy (0.476 vs. 0.472), while AIPO uses 34.3% fewer interactions. From step 50 to step 200, the average interaction count decreases by 50.4% for AIPO and 15.6% for the single-collaborator variant.
Values are averaged over 21-step windows centered on steps 50 and 100, and over steps 1–11 and 190–200 at the boundaries.
2) It makes the interaction process more structured and interpretable. By inspecting the interaction process, we can better understand what capabilities the student model lacks and how it tends to reason across different tasks.
Core contribution. Our contribution is the learning algorithm, which places no restriction on the role or implementation of the collaborators. The corrected importance sampling coefficient and the lower-bound clipping strategy (Section 3.2) let the policy learn from collaborator-generated tokens despite the off-policy discrepancy. Replacing the three specialized agents with a single general-purpose collaborator (same Qwen2.5-7B-Instruct backbone) yields comparable performance, as shown in the table below. AIPO therefore learns effectively from interaction even without the three-agent design.
W1 (part 2): Whether actively choosing when to request help is necessary
Active requests make external guidance targeted and cheap: the policy asks for help only at its own reasoning bottlenecks, so it receives state-dependent guidance, needs fewer collaborator calls as it improves, and avoids generating complete expert trajectories.
Why active requests help.
Passive guidance performs worse. LUFFY learns passively from pre-collected, rejection-sampled expert trajectories. It uses the same teacher backbone as AIPO and the same shared training configuration (Appendix B.8, Lines 1281–1286, and Table 3), so the two methods differ mainly in how external guidance enters training. As a result, AIPO outperforms LUFFY on every benchmark in Table 1, for both policy models and both collaborator choices, and converts external tokens into performance gains more efficiently (3.43 vs. 2.68 ×10⁻³ points per 10,000 external tokens, see our response to Reviewer tkop).