acceptodds
Under review as a conference paper at ICLR 2027

AIPO: Learning to Reason from Active Interaction

Abstract

Recent advances in large language models (LLMs) have demonstrated strong reasoning capabilities, largely stimulated by reinforcement learning with verifiable rewards (RLVR). However, the exploration of existing RLVR algorithms remains largely constrained by the knowledge and reasoning strategies already accessible to the policy model. Although recent methods introduce external expert demonstrations to broaden exploration, they typically rely on complete trajectory-level guidance, which can be sample-inefficient, information-sparse, and insufficiently adaptive to intermediate reasoning bottlenecks. Inspired by collaborative multi-agent systems, we propose AIPO, an enhanced reinforcement learning framework that improves LLM reasoning through active multi-agent interaction during exploration. Specifically, when encountering reasoning bottlenecks, AIPO enables the policy model to proactively consult three functional collaborative agents, namely the , , and , thereby obtaining fine-grained and state-dependent guidance during rollout. The resulting mixed-policy trajectories expose the policy to reasoning directions that may be difficult to discover through isolated on-policy exploration. To learn effectively from collaborator-provided tokens, we further introduce a corrected importance sampling coefficient together with a lower-bound clipping strategy to mitigate off-policy discrepancy and vanishing gradients. After training, the policy model reasons independently without relying on collaborative agents. Extensive experiments across mathematical, scientific, coding, and puzzle reasoning benchmarks show that AIPO consistently improves reasoning performance and generalizes across different policy models, collaborator backbones, and RLVR algorithms. Additional out-of-distribution and training-dynamics analyses indicate that active interaction broadens the policy model's , enabling it to solve additional problems independently under the same inference-time sampling budget.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, LLM Reasoning, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_a8W2OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

AIPO trains a reasoning policy to request verification, knowledge, or subproblem-solving assistance during its own rollouts. It learns from both student and collaborator tokens, using a one-hot surrogate and a gradient-preserving lower bound for external-token updates. The resulting policy is evaluated without collaborators. Experiments cover eight tasks, self and stronger collaborators, expert-guided baselines, component ablations, a 32B extension, and training-cost comparisons.

Critical strengths
  1. A novel LLM RL algorithm of AIPO is proposed via collecting verification, knowledge, or subproblem-solving assistance during its own rollouts.

  2. The empirical comparison is reasonably broad and includes LUFFY and OPSD, rather than only ordinary RL. Improvements across several task families and collaborator settings make the result more persuasive than an isolated mathematics gain.

  3. The final evaluation removes collaborators, directly testing the intended transfer to independent reasoning. The external-token ablations and the decline in low-probability external tokens provide useful evidence about the proposed learning mechanism, beyond showing that assisted rollouts are easier.

Critical weaknesses
  1. The rationale for the specific interaction design of AIPO is not sufficiently developed. While the motivation for requesting help during reasoning is understandable, it is less clear why these three forms of assistance are the appropriate choice and whether actively choosing when to request them is necessary. The existing ablations show that the components help, but do not clearly separate the benefit of active interaction from simply receiving additional guidance.

  2. The student models are relatively dated. Qwen3 is used as a collaborator rather than as the student, while the 32B extension still uses Qwen2.5. Experiments with a more recent student would help establish whether the gains persist when the policy already has stronger reasoning capabilities and may benefit less from external assistance.

  3. The larger-model experiments mainly compare against SFT and GRPO. Although LUFFY and OPSD are included in the main experiments. More recent and competitive LLM RL methods should be included in the comparison.

  4. LLM RL training is already resource-intensive, and AIPO additionally requires serving collaborator models and processing longer interaction trajectories. This adds memory and compute demands, particularly when using a larger collaborator.

Questions for authors

Please see Critical Weaknesses.

8 comments
AuthorsOpenReview ↗

Response to Reviewer a8W2 (Part 1 / 3)

We thank the reviewer for the constructive review and for recognizing the novelty of AIPO, the extensive comparisons, and the collaborator-free evaluation. We address each weakness below.

W1 (part 1): Why these three forms of assistance are the appropriate choice

The three collaborators provide complementary support, but AIPO does not depend on them: a single general-purpose collaborator achieves the same average performance (35.4 over the eight benchmarks, please see our response to Reviewer QhTy, W1).

Why three forms of assistance. 1) It provides more stable and structured supervision during training. AIPO enables the student model to improve through active interaction with external agents. The key question is therefore how to design interactions that effectively facilitate learning. Inspired by the key capabilities required in complex reasoning, we explicitly introduce three specialized agents targeting knowledge acquisition, verification, and reasoning. By decomposing the interaction into these complementary capabilities, AIPO provides clearer and more structured learning signals, reducing the difficulty of deciding what kind of assistance is needed. This mainly improves the training process, making learning more stable and interaction-efficient without being essential to the final capability of the student model. The training dynamics below support this effect (detailed figures are added in Appendix C.6). Around step 100, AIPO reaches 0.453 batch task accuracy, compared with 0.260 for the single-collaborator variant, while using 25.2% fewer interactions per batch. Around step 200, both reach comparable accuracy (0.476 vs. 0.472), while AIPO uses 34.3% fewer interactions. From step 50 to step 200, the average interaction count decreases by 50.4% for AIPO and 15.6% for the single-collaborator variant.

StepBatch Task Accuracy ↑ (Single / AIPO)Failed Tasks per Batch ↓ (Single / AIPO)Interactions per Batch ↓ (Single / AIPO)
10.152 / 0.144169.4 / 154.1721.7 / 249.7
500.251 / 0.270127.7 / 126.04,737.0 / 5,299.0
1000.260 / 0.453134.5 / 84.74,950.4 / 3,701.3
2000.472 / 0.47656.5 / 46.23,997.0 / 2,627.2

Values are averaged over 21-step windows centered on steps 50 and 100, and over steps 1–11 and 190–200 at the boundaries.

2) It makes the interaction process more structured and interpretable. By inspecting the interaction process, we can better understand what capabilities the student model lacks and how it tends to reason across different tasks.

Core contribution. Our contribution is the learning algorithm, which places no restriction on the role or implementation of the collaborators. The corrected importance sampling coefficient and the lower-bound clipping strategy (Section 3.2) let the policy learn from collaborator-generated tokens despite the off-policy discrepancy. Replacing the three specialized agents with a single general-purpose collaborator (same Qwen2.5-7B-Instruct backbone) yields comparable performance, as shown in the table below. AIPO therefore learns effectively from interaction even without the three-agent design.

W1 (part 2): Whether actively choosing when to request help is necessary

Active requests make external guidance targeted and cheap: the policy asks for help only at its own reasoning bottlenecks, so it receives state-dependent guidance, needs fewer collaborator calls as it improves, and avoids generating complete expert trajectories.

Why active requests help.

  • Targeted guidance. Collaborator feedback is conditioned on the policy's current partial trajectory, and about 10% of training samples need no interaction at all (Appendix C.7).
  • Lower cost. Per-batch collaborator calls decrease by about 30% from step 50 to step 200 as the policy internalizes the guidance (Figure 6). AIPO with a same-scale Qwen2.5-7B-Instruct collaborator matches LUFFY with the larger Qwen3-30B-A3B teacher (35.4 average) at about half the GPU hours (166 vs. 310, Table 9).

Passive guidance performs worse. LUFFY learns passively from pre-collected, rejection-sampled expert trajectories. It uses the same teacher backbone as AIPO and the same shared training configuration (Appendix B.8, Lines 1281–1286, and Table 3), so the two methods differ mainly in how external guidance enters training. As a result, AIPO outperforms LUFFY on every benchmark in Table 1, for both policy models and both collaborator choices, and converts external tokens into performance gains more efficiently (3.43 vs. 2.68 ×10⁻³ points per 10,000 external tokens, see our response to Reviewer tkop).

AuthorsOpenReview ↗

Response to Reviewer a8W2 (Part 2 / 3)

W2: The student models are relatively dated

We added experiments with more recent LLMs: Qwen3-8B in non-thinking mode (Qwen3-8B-NoThinking) as the policy model. The results are shown in the table below and added in Appendix C.7. Within each collaborator setting, AIPO outperforms all baselines on every benchmark.

Averaged over the eight benchmarks, AIPO with a same-scale Qwen3-8B-NoThinking collaborator improves the original model from 40.3 to 46.6. This is 1.3 points above the strongest baseline that does not use a larger collaborator, Dr.GRPO (45.3), and 1.9 points above LUFFY with the same collaborator (44.7). With the larger Qwen3-30B-A3B-Instruct-2507 collaborator, AIPO reaches 47.9, 1.3 points above LUFFY (46.6). AIPO with the same-scale collaborator already matches LUFFY with the larger collaborator (46.6 vs. 46.6). These results suggest that a policy with stronger reasoning ability can still benefit from external assistance.

MethodsAIME24 (Avg@16)AIME25 (Avg@16)MATH500 (Avg@4)LMB (Avg@4)GPQA-D (Avg@4)MBPP (Avg@4)LCB (Avg@4)RG (Avg@4)
Qwen3-8B-NoThinking
Original29.1±0.120.9±0.387.4±0.222.4±0.444.4±0.172.8±0.328.7±0.116.4±0.2
GRPO39.7±0.430.6±0.389.9±0.125.1±0.248.6±0.574.5±0.130.7±0.420.9±0.3
PRIME38.8±0.630.1±0.489.0±0.323.0±0.546.5±0.772.1±0.429.2±0.618.8±0.3
Dr.GRPO40.0±0.330.9±0.290.1±0.125.2±0.148.8±0.474.7±0.131.0±0.321.4±0.1
Qwen3-8B-NoThinking ↔\leftrightarrow Qwen3-8B-NoThinking
SFT37.1±0.529.1±0.487.6±0.322.7±0.544.7±0.773.0±0.428.9±0.616.6±0.3
OPSD39.3±0.430.5±0.389.3±0.124.1±0.348.7±0.573.7±0.229.8±0.419.0±0.3
LUFFY40.1±0.630.2±0.589.2±0.324.5±0.449.2±0.674.1±0.330.1±0.520.4±0.4
AIPO42.2±0.232.7±0.190.9±0.126.0±0.151.4±0.375.4±0.131.7±0.222.3±0.1
Qwen3-8B-NoThinking ↔\leftrightarrow Qwen3-30B-A3B-Instruct-2507
SFT39.0±0.530.6±0.488.8±0.323.9±0.545.2±0.773.3±0.429.3±0.617.2±0.3
OPSD41.7±0.432.0±0.390.3±0.125.4±0.350.3±0.574.5±0.230.7±0.421.1±0.3
LUFFY42.4±0.632.6±0.491.1±0.226.1±0.451.5±0.475.4±0.331.7±0.522.0±0.4
AIPO44.0±0.633.6±0.491.7±0.228.2±0.452.4±0.476.2±0.333.3±0.523.9±0.4

W3: The 32B experiments should include more competitive RL baselines

Thank you for the suggestion. We added LUFFY and OPSD to the Qwen2.5-32B-Instruct experiments, using the same Qwen2.5-32B-Instruct collaborator. The results are shown in the table below where AIPO outperforms all baselines on every benchmark.

Averaged over the eight benchmarks, AIPO reaches 57.8, compared with 55.6 for LUFFY, 53.5 for GRPO, 52.2 for OPSD, and 50.7 for SFT. This is 2.2 points above the strongest baseline (LUFFY) and 9.7 points above the original model (48.1), indicating that the advantage of AIPO persists at the 32B scale.

MethodsAIME24 (Avg@16)AIME25 (Avg@16)MATH500 (Avg@4)LMB (Avg@4)GPQA-D (Avg@4)MBPP (Avg@4)LCB (Avg@4)RG (Avg@4)
Qwen2.5-32B-Instruct ↔\leftrightarrow Qwen2.5-32B-Instruct
Original29.9±0.428.6±0.486.4±0.327.8±0.562.6±0.580.6±0.540.2±0.428.6±0.4
SFT32.6±0.431.3±0.488.3±0.230.3±0.465.3±0.583.6±0.442.9±0.431.1±0.4
OPSD34.3±0.533.1±0.588.4±0.232.0±0.567.0±0.485.3±0.544.6±0.432.8±0.5
GRPO35.7±0.534.5±0.588.6±0.233.5±0.568.4±0.486.7±0.546.0±0.434.2±0.5
LUFFY37.4±0.637.1±0.588.9±0.236.2±0.570.5±0.588.5±0.447.2±0.439.1±0.5
AIPO39.1±0.439.7±0.489.3±0.238.9±0.472.6±0.490.2±0.348.4±0.444.0±0.4
AuthorsOpenReview ↗

Response to Reviewer a8W2 (Part 3 / 3)

W4: Serving collaborators adds memory and compute, particularly with a larger collaborator

We agree that serving and querying collaborators adds training cost. But serving collaborators adds a moderate, training-only overhead. AIPO costs 11% more GPU hours than GRPO with a same-scale collaborator and about the same as teacher-based baselines with a larger one, outperforms GRPO under a matched compute budget, and needs no collaborator at inference.

First, the overhead is moderate with a same-scale collaborator, and with a larger collaborator it is on par with teacher-based baselines that use the same model. With a same-scale Qwen2.5-7B-Instruct collaborator, AIPO adds 11% total GPU hours (149 → 166) and 7% peak memory (286 → 307 GB) over GRPO, while raising the average from 33.3 to 35.4 (Table 9). This matches LUFFY with the larger Qwen3-30B-A3B teacher (35.4) at about half its GPU hours (166 vs. 310). With Qwen3-30B-A3B as the collaborator, the total cost of AIPO is comparable to OPSD (329 vs. 326 GPU hours) and LUFFY (329 vs. 310, including offline trajectory generation), while its average is 2.8 and 1.7 points higher, respectively.

Second, the gains persist under matched compute. With an identical GPU-hour budget, AIPO outperforms GRPO in all four domains (Table 10).

Third, the overhead is adaptive and limited to training. The policy decides when to query collaborators, and per-batch calls decrease by about 30% from step 50 to step 200 (Figure 6). At inference, the policy reasons alone, with no additional memory, latency, or API cost.


We hope our responses could address your concerns. If you have any further questions, please do not hesitate to let us know.

Reviewer_tkopOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AIPO lets a policy request verification, knowledge, or subproblem-solving feedback during RL rollouts. It learns from both policy and collaborator tokens using a probability-based external-token weight and gradient floor. Collaborators are removed at inference. Evaluation covers mathematical, scientific, coding, and puzzle reasoning.

Critical strengths
  1. Training-time assistance produces gains when the policy later reasons alone. The matched-GPU-hour comparison and transfer across RL algorithms support a useful training recipe.
  2. The same-scale collaborator and external-token masking experiments are informative. Verify-only and reasoning-only gains also show that the effect does not require the reference-answer-conditioned knowledge agent.
Critical weaknesses
  1. Proposition A.1 bounds the surrogate expectation under the old policy, whereas observed external tokens are sampled from collaborators. This does not bound the implemented estimator's error, and the gradient floor is outside the analysis. The importance-sampling justification is consequently weaker than claimed.
  2. Comparisons with SFT, OPSD, and LUFFY change both the guidance mechanism and the trajectories received. There is no budget-matched fixed-call control using the same collaborators, so the specific benefit of learned interaction timing remains unclear.
Questions for authors
  1. How does Proposition A.1 apply to collaborator-sampled tokens? A comparison with reward-weighted cross-entropy on identical traces would also help.
  2. How does fixed-frequency assistance perform with the same collaborator-token budget?
3 comments
AuthorsOpenReview ↗

Response to Reviewer tkop (Part 1 / 2)

We thank the reviewer for the careful and technically precise review, and for recognizing our method and experimental results. We address each concern and question below.

W1 & Q1: Whether Proposition A.1 applies to collaborator-sampled tokens

We agree that Proposition A.1 does not bound the implemented estimator, and we will add discussion and theories about it. Our update is a sign-preserving reweighting of the exact importance-sampling (IS) gradient, and we will present it as an IS-inspired surrogate rather than an exact IS correction.

Corrected bound for the implemented estimator. External tokens are sampled from πϵ\pi_\epsilon, so the implemented second term has expectation ∑xπϵ(x)πθ(x)A~\sum_x \pi_\epsilon(x)\pi_\theta(x)\tilde{A}, whereas the exact IS term is ∑xπθ(x)A~\sum_x \pi_\theta(x)\tilde{A}. Repeating the proof of Proposition A.1 with πθold\pi_{\theta_{\text{old}}} replaced by πϵ\pi_\epsilon bounds their gap by Amax⁡(η(πθ)+2dTV(πθ,πϵ))A_{\max}(\eta(\pi_\theta) + 2 d_{\mathrm{TV}}(\pi_\theta, \pi_\epsilon)).

Each token's gradient keeps the direction of the exact IS gradient. For a sampled external token xx, the exact IS gradient is (πθ(x)/πϵ(x))A~∇θlog⁡πθ(x)(\pi_\theta(x)/\pi_\epsilon(x))\tilde{A}\nabla_\theta \log \pi_\theta(x), and our implemented gradient is max⁡(πθ(x),ω)A~∇θlog⁡πθ(x)\max(\pi_\theta(x), \omega)\tilde{A}\nabla_\theta \log \pi_\theta(x) (Equation (7)). Both weights are positive, so only the magnitude of each token's update differs.

The gradient floor is a deliberate departure from IS, justified empirically. It prevents vanishing gradients for external tokens that the policy initially finds unlikely. Removing it reduces performance by 1.0–4.3 points (Table 2, w/o CS). The fraction of external tokens with πθ<ω\pi_\theta < \omega falls from 21% at initialization to 7% by step 100 (Appendix C.8), consistent with the policy increasingly agreeing with collaborator tokens.

The surrogate is effective in practice. Replacing our coefficient with the standard ratio reduces performance on all eight benchmarks by 0.8–3.0 points (Table 2, w/o IS), and AIPO outperforms RWCE on identical traces by 1.9–2.7 points (table below, added in Appendix C.9).

Qwen2.5-7B-Inst ↔\leftrightarrow Qwen2.5-7B-InstAIME24AIME25MATH500LMBGPQA-DMBPPLCBRG
AIPO26.5±0.221.3±0.180.5±0.114.9±0.141.7±0.362.7±0.219.2±0.216.0±0.1
RWCE (same traces)24.2±0.318.7±0.478.5±0.212.2±0.339.6±0.460.2±0.117.3±0.513.6±0.3

[1] Shao, Zhihong, et al. "DeepSeekMath: Pushing the limits of mathematical reasoning in open language models." arXiv preprint arXiv:2402.03300 (2024).
[2] Lanchantin, Jack, et al. "Bridging offline and online reinforcement learning for LLMs." arXiv preprint arXiv:2506.21495 (2025).

[3] Tajwar, Fahim, et al. "Preference fine-tuning of LLMs should leverage suboptimal, on-policy data." arXiv preprint arXiv:2404.14367 (2024).

AuthorsOpenReview ↗

Response to Reviewer tkop (Part 2 / 2)

W2 & Q2: Budget-matched control with a fixed number of collaborator calls

We agree that a comparison under a fixed external-token budget is valuable. The external-token budget is hard to match directly, because the methods consume external tokens through different mechanisms. We therefore compare the gain per external token, and AIPO is the most efficient: 3.43×10−33.43 \times 10^{-3} points per 10,000 external tokens, compared with at most 2.68×10−32.68 \times 10^{-3} for the baselines.

Why the budget cannot be matched directly. Fixing the external-token count in advance would require changing the methods themselves: 1) AIPO calls collaborators adaptively. The policy decides when to call a collaborator, so the number of calls rises early in training and then declines as the policy internalizes collaborator guidance (Figures 5 and 6), and about 10% of samples use no collaborator at all (Appendix C.7). 2) The baselines consume external tokens in fixed forms. SFT and LUFFY learn from complete pre-collected expert trajectories, and OPSD queries the teacher distribution on every student-generated token.

Gain per external token as a proxy for performance under the same budget:

  • For each method, we count the total number of external tokens consumed during training, NextN_{\text{ext}}, and the improvement in average benchmark performance over the untrained base model, ΔS\Delta S.
  • We report the gain per 10,000 external tokens, ΔS/Next×104\Delta S / N_{\text{ext}} \times 10^4.

The gains are not explained by a larger token budget: AIPO consumes 13% more external tokens than LUFFY but achieves a 45% larger gain (8.1 vs. 5.6 points), as shown in the table below.

Qwen2.5-7B-Inst ↔\leftrightarrow Qwen2.5-7B-InstAvg. Perf.SNext (104)S per 104 tokens (10-3)
SFT30.02.71929.21.40
OPSD32.45.12197.42.32
LUFFY32.95.62089.82.68
AIPO35.48.12359.83.43

We hope our responses could address your concerns. If you have any further questions, please do not hesitate to let us know.

AuthorsOpenReview ↗

Follow-up of Rebuttal

Dear Reviewer tkop,

Thank you for your positive assessment of our work and for your careful and technically precise review. As the discussion period is about to close, we would like to highlight our response above. In response to your reviews, we have added a bound that directly covers the implemented estimator, shown that each external token's update keeps the sign of the exact importance-sampling gradient, and shown that AIPO achieves the highest gain per external token when the external-token budget cannot be matched directly.

Could you please let us know whether these additions have addressed your concerns regarding the theoretical justification and the budget-matched comparison? If so, we would appreciate it if you could consider raising your score to further support our work.

If any concerns remain, we would be glad to clarify them further while the discussion period is still open.

Sincerely,

Authors

Reviewer_QhTyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper provides a new implementation of mixed policy RL for solving the exploration limitation problem of on-policy RL. The direction is well motivated and makes a lot of sense. The authors propose an active interaction approach to solve this problem, where the multi-agent interaction, including the knowledge agent, verify agent and reasoning agent, serves as the exploration part. In addition, experiments show some improvements against existing methods like LUFFY. However, my main concern is that the proposed method is not a principled approach and is kind of outdated for current LLMs. Specifically, the approach is more like injecting a human inductive bias of outside resources. However, current LLMs as agents have already implemented such functionalities. I cannot think of a naive way of adapting such a method to incorporate into long-horizon agentic RL or other agentic scenarios. Also, I would say some of the experiments look suspicious. For example, the Qwen3 30B-A3B has scored 61.3 in AIME 25 but with this model as the teacher, SFT in your experiments only score 18.8, which seems incorrect.

Critical strengths

This paper addresses a real limitation of on-policy RLVR, bounded exploration, with a clean and practical idea. The empirical evaluation is broad and consistent across benchmarks, model families, RLVR algorithms, and collaborator settings. The DeepMath training-dynamics analysis provides convincing evidence that AIPO expands the effective exploration boundary, and the efficiency analysis and limitations are discussed transparently.

Critical weaknesses

See above

Questions for authors

How to incorporate your work into scenarios like agentic RL for coding or swe?

5 comments
AuthorsOpenReview ↗

Response to Reviewer QhTy (Part 1 / 2)

We thank the reviewer for the positive assessment of the problem, the idea, and the breadth of our evaluation. We address each concern below.

W1: AIPO injects a human inductive bias through hand-designed external resources

The three collaborators turn capabilities the policy already has into explicit interactions during training, so the policy can strengthen them by learning from these interactions. The gains come from learning through interaction rather than from this specific design: with a single general-purpose collaborator, AIPO performs comparably (35.40 vs. 35.35 averaged over the eight benchmarks).

Why three collaborators. We agree that modern agents already verify, retrieve, and delegate. The three collaborators make these existing capabilities explicit during training: the policy decides when to verify a conclusion, retrieve knowledge, or delegate a sub-problem, receives targeted feedback, and learns from it. Interaction therefore strengthens capabilities the policy already has rather than injecting new hand-designed ones, and all collaborators are removed at test time.

  • The three collaborators are complementary. Any single collaborator still outperforms the baseline, with the Knowledge Agent contributing most on knowledge-intensive science benchmarks and the Verify and Reasoning Agents more effective elsewhere (Figure 3).
  • The policy internalizes what it learns. Masking collaborator tokens during training reduces performance by 0.8–4.1 points (Table 2), and the masked variant never reduces the number of tasks it fails to solve (Figure 5).

Without this design. Our contribution is the learning algorithm, which places no restriction on the role or implementation of the collaborators. The corrected importance sampling coefficient and the lower-bound clipping strategy (Section 3.2) let the policy learn from collaborator-generated tokens despite the off-policy discrepancy. Replacing the three specialized agents with a single general-purpose collaborator (same Qwen2.5-7B-Instruct backbone) yields comparable performance, as shown in the table below. AIPO therefore learns effectively from interaction even without the three-agent design. We add these results in Appendix C.6.

Qwen2.5-7B-Inst ↔\leftrightarrow Qwen2.5-7B-InstAIME24AIME25MATH500LMBGPQA-DMBPPLCBRG
AIPO (three specialized collaborators)26.5±0.221.3±0.180.5±0.114.9±0.141.7±0.362.7±0.219.2±0.216.0±0.1
AIPO (single general-purpose collaborator)27.1±0.320.8±0.279.8±0.315.6±0.240.3±0.164.1±0.418.7±0.316.8±0.3

W2/Q: Extension to long-horizon agentic RL and other agentic settings

We expect AIPO to extend naturally to agentic RL. In agentic RL, the model's generation consists of a reasoning process and action selection, and AIPO covers both: Reasoning process. AIPO applies directly, with the policy querying collaborators during reasoning as in our current setting. Action selection. An additional prompt lets the policy request hints from collaborators about which action to take, and the resulting trajectories are optimized with our proposed objective.

The table below reports AIPO in the tool-integrated reasoning (TIR) setting (also add in Appendix C.8), where the model can invoke a Python interpreter code tool with up to 3 per rollout and the same training data as in the main experiments. AIPO remains effective under TIR, which suggests that it is applicable to agentic RL.

Qwen2.5-7B-Inst ↔\leftrightarrow Qwen2.5-7B-Inst (TIR)AIME24AIME25MATH500LMB
GRPO28.9±0.525.1±0.485.7±0.319.5±0.7
AIPO33.3±0.329.8±0.289.4±0.525.7±0.3
AuthorsOpenReview ↗

Response to Reviewer QhTy (Part 2 / 2)

Q: SFT scores only 18.8 on AIME25

The 18.8 in Table 1 is the AIME25 score of Qwen2.5-7B-Instruct after SFT on rejection-sampled teacher trajectories (Appendix B.8). This result is consistent with those reported in [1] and [2], which achieve 18.24 and 19.58 on AIME25, respectively. Even when using the stronger teacher model DeepSeek-R1-671B, the method in [3] achieves only 11.25 on AIME25, below the result of our implementation. We believe that the performance of SFT is limited by the student model’s capabilities or by the gap between the student and teacher models, both of which may make it difficult for the student to learn from the teacher through SFT. However, this problem is out of research scope. The teacher, Qwen3-30B-A3B-Instruct-2507, obtains 64.0 Avg@16 on AIME25 under our protocol (Table 4, Appendix C.1), consistent with its officially reported 61.3, so our evaluation does not underestimate the teacher. We state this explicitly in the caption of Table 1.

[1] Yu, Zhaochen, et al. "Demystifying reinforcement learning in agentic reasoning." arXiv preprint arXiv:2510.11701 (2025).
[2] Huang, Huipeng, and Hongxin Wei. "Tail-Aware Top-$ k $ On-Policy Distillation." arXiv preprint arXiv:2608.14728 (2026).
[3] Luo, Yashuo, et al. "From Solver Trajectories to Teaching Trajectories: Cognition-Aligned Reasoning Distillation." arXiv preprint arXiv:2610.36445 (2026).


We hope our responses could address your concerns. If you have any further questions, please do not hesitate to let us know.

Thank you for the additional experiments in agentic scenarios and the clarification regarding the AIME scores. Your results show that without any injected inductive bias the naive multi-agent approach work even better, it leads to a huge change of your paper claims and my evaluation of your novelty. Therefore, I keep my score as is.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.