acceptodds
Under review as a conference paper at ICLR 2027

InverseHarness: Internalizing Agentic Reasoning through Reinforcement Learning

Abstract

Reinforcement learning (RL) improves the reasoning capability of large language models (LLMs) by searching and then memorizing successful trajectories, but progress can stall when effective strategies are rarely discovered due to the difficulty of problems through native sampling. Agentic harnesses can elicit stronger reasoning through external orchestration, yet their executions are not directly usable as standalone training trajectories. We introduce HAIR (Harness-Assisted Internalization via Reinforcement Learning), which combines an agentic harness with InverseHarness to reconstruct successful executions into self-contained trajectories while preserving the current policy’s generated reasoning. Interleaved with standard RL, this process targets problems where native rollouts fail and regenerates training trajectories from the evolving policy. On mathematical reasoning with Intern-S2-Preview at 35B and 397B, HAIR outperforms GRPO across most evaluated benchmarks under matched backbones, training corpora, and optimizer steps. At 397B, pass@1 increases from 55.5 to 68.8 on IMO-Proof-Bench and from 29.2 to 38.4 on PhD-Paper. Across rounds, harnessed performance improves with the policy, while training on successful executions improves standalone reasoning, supporting a closed loop of self-improvement. Further analysis shows that generating trajectories from the current policy matters beyond correctness alone. These findings establish harness-assisted training as a promising route to self-improvement beyond native sampling.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language model, reinforcement learning, self improvement

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_kgbBOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes HAIR, which uses a reasoning harness to solve difficult problems and reconstructs successful executions into standalone trajectories for supervised updates interleaved with GRPO.

Critical strengths

A compelling approach to learning from problems that native sampling fails to solve.

Critical weaknesses

See Questions For Authors

Questions for authors
  1. I am curious whether retaining trial-and-error and self-verification changes the model’s response length, especially on easier problems. Have the authors observed any interesting trends?

  2. Could the authors share the length limit used for SFT trajectories and how particularly long reconstructed traces are handled?

5 comments

Thank you for your thoughtful review and constructive suggestions. We address your concerns below.

AuthorsOpenReview ↗

1. Effect on response length

Thank you for this thoughtful question. We indeed observe that HAIR increases response length relative to GRPO. This trend is evident in both the synthesized SFT data and the training rollouts from the final model snapshots. The latter are summarized below:

StatisticGRPOHAIR
Mean response length (tokens)41,72347,206
Responses reaching the generation length limit14.4%33.8%

Our inspection of the trajectories suggests that additional self-verification accounts for part of this increase: the model spends more tokens checking its intermediate reasoning and candidate solutions. This trend is consistent with HAIR’s primary objective of improving performance on challenging problems, where retaining trial-and-error and self-verification can support more thorough reasoning.

AuthorsOpenReview ↗

2. SFT length limit and handling of long trajectories

Our SFT trajectory length limit is 128k tokens, compared with a total length limit of 64K tokens during RL. We empirically found that allowing trajectories longer than 64K tokens in SFT yields greater performance gains during subsequent RL.

Our interpretation is that, because HAIR trajectories are sampled from the current policy, the successful trajectory primarily teaches the model how to structure its reasoning policy, including coordinate trial-and-error and self-verification, rather than memorize entire trajectories. Longer trajectories can therefore provide useful training signals that translate into improved performance under the shorter RL response budget.

Regarding particularly long reconstructed traces, none of the synthesized trajectories exceed 128k tokens, as their length is bounded by the context limit of the final policy completion step.

Reviewer_5E5qOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces HAIR, which alternates standard GRPO with harness-assisted supervised updates to improve standalone mathematical reasoning. For problems that the current policy fails to solve within a finite native sampling budget, the authors use the same policy for multiple rounds of solving, verification, and repair. InverseHarness then reconstructs successful executions into self-contained reasoning prefixes. The current policy continues these prefixes, and responses that pass correctness filtering are used for supervised training. The authors evaluate the framework at two model scales, report improvements over GRPO on proof-style tasks, and investigate the training gains through component ablations, a trajectory-provenance comparison, and multi-stage analyses.

Critical strengths
  1. The lack of successful training examples for difficult problems remains an important challenge in training reasoning models. The paper explores improving standalone reasoning through external orchestration, which has clear practical value.
  2. The method improves performance across multiple mathematical benchmarks at two model scales, providing empirical support for the feasibility of generating training data from harness executions.
Critical weaknesses
  1. HAIR largely follows established self-training approaches, such as STaR[1], ReST-MCTS*[2], and rStar-Math[3], which use successful reasoning trajectories generated through enhanced search for policy improvement. Moreover, the paper lacks comparisons with simpler trajectory rewriting or distillation methods to establish the necessity of its design.
  2. Harness evaluation is limited to Intern-S1-MO and its simplified variant, without testing adaptability to other orchestration structures, intermediate states, or interaction patterns. The research-level proof benchmark PhD-Paper contains only 26 problems, and repeated sampling cannot compensate for limited coverage of independent problems. Consequently, both generality across harnesses and effectiveness on a broader range of research-level mathematical problems remain insufficiently supported.
  3. HAIR adds problem screening, multi-round solving and verification, editing, continuation, and judging, but the main experiments match only optimizer steps and do not quantify end-to-end training costs. The inference-time generated-token comparison cannot substitute for a training-cost analysis, so the current results do not establish an advantage under the same total computational budget, leaving the method’s practical cost-effectiveness difficult to assess. [1]STaR: Self-Taught Reasoner Bootstrapping Reasoning With Reasoning. [2]ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search. [3]rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking.
Questions for authors

1.Why do task-specific gains differ across model scales? The smaller model improves substantially on APEX-2025, whereas the larger model shows no improvement in pass@1 and a slight decrease in pass@4.

  1. How should the decline in subsequent standalone solving be interpreted? Among problems solved by the harness, the fraction subsequently solved by the standalone policy under pass@8 falls from 46.0% in the first stage to 19.8% in the last, while the problem pool also changes across stages. Are there existing statistics that help distinguish increasingly difficult problems from limitations in the model’s ability to absorb the training trajectories?

  2. What explains the initial decline and subsequent recovery when using trajectories from another source? Could the authors identify the alternative source model, along with the main differences in trajectory length and reasoning style relative to current-policy trajectories? Are there observations of losses or training dynamics that help explain the initial decline and subsequent recovery?

  3. What motivates the supervised-update frequency? Why is one supervised update inserted after every 40 GRPO iterations, and was this choice based on experience or preliminary experiments? If observations at other frequencies are available, could the authors explain whether more or less frequent supervised updates change training effectiveness?

17 comments

Thank you for your thoughtful review and constructive suggestions. We address your concerns below.

AuthorsOpenReview ↗

1. Relationship to prior self-training methods

Thank you for highlighting these related self-training approaches. We will add these works to the next version of our draft. HAIR’s central distinction is that it strengthens policy improvement through harness-generated experience under the same supervision setting as standard outcome-rewarded RL, without requiring reference answers, complete proofs, or stronger teacher model as additional supervision. It demonstrates the effectiveness of this approach at tasks and scales relevant to frontier LLM training.

  1. Solution discovery without answer-conditioned rationalization or reference derivations. STaR supplies the correct answer when the model fails and asks it to generate a supporting rationale. ReST-MCTS* initializes its science-domain value model using SciInstruct questions paired with correct step-by-step solutions, relying on existing solution-level supervision to bootstrap search guidance. HAIR instead discovers solutions through harness-driven inference, without providing a reference answer as a rationalization hint. A core strength of HAIR is that policy improvement requires no reference information and operates in exactly the same setting as standard outcome-rewarded RL. This is central to its role as a stronger policy improvement operator for self-improvement.

  2. Self-improvement without a stronger teacher. HAIR uses the evolving policy itself to generate harness-assisted solutions and internalizes these improvements back into the same policy. In contrast, rStar-Math bootstraps its initial training trajectories using DeepSeek-Coder-V2-Instruct (236B) before proceeding to self-evolution with smaller policy models. HAIR instead obtains its solution trajectories through harness-enhanced execution of the current policy, without requiring a stronger solution-generating teacher or pre-existing reference derivations for those problems.

  3. Advanced proof tasks and large model scales. HAIR evaluates complete proof generation across competition mathematics, undergraduate mathematics, doctoral qualifying examinations, and research-level problems drawn from published papers. We demonstrate improvements at both 35B and 397B, with the 397B model achieving competitive performance against leading open-source models. These results demonstrate HAIR’s effectiveness at tasks and scales relevant to frontier LLM training.

  4. Internalization into standalone reasoning. HAIR’s primary objective and evaluation concern the backbone’s direct, harness-free reasoning capability. By comparison, rStar-Math’s headline results rely on test-time MCTS guided by a process preference model. Its Qwen2.5-Math-7B achieves 90.0 on MATH and 53.3 on AIME24 with search, versus 78.4 and 26.7 in its separately reported greedy-decoding evaluation. HAIR explicitly targets absorbing the benefits of enhanced inference into the policy itself, so that its standalone gains do not require the generation-time harness at deployment.

[1]STaR: Self-Taught Reasoner Bootstrapping Reasoning With Reasoning.

[2]ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search.

[3]rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking.

AuthorsOpenReview ↗

2. Comparisons with simpler trajectory rewriting or distillation methods

Thank you for raising this important point. We address this concern from two complementary perspectives: simplifying the harness and replacing harness-based generation with reference-conditioned distillation.

  1. Simplifying the harness substantially reduces downstream gains. In our “Weakening the harness” setting of ablation study (line 438), we replace Intern-S1-MO with a simplified solve–verify loop harness. This variant retains only 25%, 56%, and 11% of HAIR’s gains over GRPO on PhD-Paper, IMO-Proof, and QE, respectively. The simplified harness solves fewer challenging problems, reducing the availability of valuable training trajectories. Thus, the richer harness serves a concrete purpose: it enables successful reasoning on difficult problems that a simpler generation process struggles to solve.

  2. Reference-conditioned reconstruction requires complete solutions and does not necessarily produce better training signals. A core strength of HAIR is that policy improvement requires no reference information and operates in exactly the same setting as standard outcome-rewarded RL. HAIR improves the current policy through harness-generated experience, without relying on reference answers or complete proofs as additional supervision. This is central to its role as a stronger policy improvement operator for self-improvement. In contrast, constructing training data from clean proofs or asking the student to reconstruct reasoning from correct solutions introduces reference information beyond this setting.

We nevertheless evaluated reference-conditioned distillation on difficult queries with available ground truth using HDPO [1], which conducts hybrid GRPO + privileged self-distillation.

MethodIMO-Proof pass@1IMO-Proof pass@4UGD pass@1UGD pass@4QE pass@1QE pass@4PhD-Paper pass@1PhD-Paper pass@4
35B baseline19.030.845.565.523.347.813.220.2
GRPO38.648.368.883.046.968.920.729.8
HDPO30.245.267.181.140.356.618.926.9
HAIR42.553.369.685.055.076.724.336.5

HDPO underperforms GRPO across all four benchmarks, whereas HAIR consistently improves both pass@1 and pass@4. These results demonstrate that reference-conditioned distillation does not reproduce the benefits of harness-generated training data in our setting.

Recent studies provide further evidence that correct reference solutions do not automatically yield effective reasoning supervision. Kaur et al. [2] find that privileged-context self-distillation can degrade thinking models by suppressing reasoning branches and self-correction. Kim et al. [3] report that fine-tuning on correct, solution-guided trajectories can substantially degrade mathematical reasoning, and associate self-distillation failures with suppressed uncertainty expression and poorer generalization.

[1] HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillation.

[2] Kaur et al. Rethinking On-Policy Self-Distillation for Thinking Models. 2026.

[3] Kim et al. Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs? COLM 2026.

Reviewer_i21SOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes HAIR, a framework designed to address the difficulty of obtaining successful reasoning trajectories through native sampling in reinforcement learning. HAIR employs an agentic harness for iterative problem solving, verification, and refinement, and uses InverseHarness to reconstruct successful multi-turn executions into standalone reasoning trajectories. These trajectories are incorporated into training through alternating SFT and GRPO updates. Experiments on 35B and 397B models demonstrate improvements over GRPO across several mathematical reasoning benchmarks.

Critical strengths
  1. The internalization of harness-assisted agentic trajectories is a topic worthy of research. Math reasoning is a good starting point because it rarely involves interaction with the real world, and the harness’s roles in “planning, verification, and scheduling” can be internalized by LLMs.
  2. Strong empirical results and useful analyses. The method achieves notable improvements across multiple mathematical reasoning benchmarks at two model scales. Comparisons between the harness and Best-of-N sampling, component ablations, and trajectory provenance analyses provide useful evidence supporting the effectiveness of the proposed framework.
Critical weaknesses
  1. I am more curious about how tasks involving interaction with the real world—such as computer use—should be internalized. This paper claims to investigate “can improvements induced by such an external operator be internalized into the model itself,” but it lacks a discussion of such tasks.
  2. Unclear benefits of harness-generated trajectories relative to their computational cost. It remains unclear whether preserving the exploration and refinement processes in harness-generated trajectories provides substantial benefits over simpler data construction approaches. Although InverseHarness removes some redundancy, the reconstructed trajectories may still contain unnecessary exploration steps and redundant reasoning. Similar training data could potentially be constructed through simpler methods, such as using clean mathematical proofs or having the student model reconstruct its own reasoning trajectories from complete correct proofs, followed by the same alternating SFT–GRPO training procedure. Comparisons with these alternatives are important to justify the substantial computational cost of harness-based data generation.
  3. Limited methodological novelty over related work. Prior works such as AgentArk [1], Back-Translated Traces [2], and Chain-of-Agents [3], among potentially other related studies, also generate high-quality data through agentic reasoning and distill it into a single model. While the specific implementations differ, these methods share a similar high-level idea with HAIR. The paper should more clearly discuss its technical distinctions and the specific advantages of its design over existing approaches.

[1]Luo, Yinyi, et al. "Agentark: Distilling multi-agent intelligence into a single llm agent." arXiv preprint arXiv:2602.03955 (2026). [2]Huang, Xingyue, et al. "Distilling Tool Knowledge into Language Models via Back-Translated Traces." arXiv preprint arXiv:2506.19171 (2025). [3]Li, Weizhen, et al. "Chain-of-agents: End-to-end agent foundation models via multi-agent distillation and agentic rl." arXiv preprint arXiv:2508.13167 (2025).

Questions for authors

please see my summarized weakness.

4 comments

Thank you for your thoughtful review and constructive suggestions. We address your concerns below.

AuthorsOpenReview ↗

1. Internalization in tasks involving interaction with an external environment

Thank you for raising this important question. We agree that tasks involving interaction with external environments, such as computer use, deserve further discussion. Our work focuses on reasoning capability as a bottleneck for policy improvement during RL. We therefore study self-contained reasoning harnesses in reasoning-intensive domains such as mathematics.

Extending internalization to interactive tasks is a promising direction, but it introduces additional challenges. In such settings, internalization could involve learning to interpret observations, predict the consequences of actions, and use feedback from the environment to guide subsequent decisions. The goal would be to internalize the capabilities supported by an external operator while retaining the interactions needed to complete the task. Our current study does not evaluate this setting, and we leave it to future work.

AuthorsOpenReview ↗

2. Benefits of harness-generated trajectories relative to simpler alternatives

Thank you for raising this important point. We address this concern from two complementary perspectives: simplifying the harness and replacing harness-based generation with reference-conditioned distillation.

  1. Simplifying the harness substantially reduces downstream gains. In our “Weakening the harness” setting of ablation study (line 438), we replace Intern-S1-MO with a simplified solve–verify loop harness. This variant retains only 25%, 56%, and 11% of HAIR’s gains over GRPO on PhD-Paper, IMO-Proof, and QE, respectively. The simplified harness solves fewer challenging problems, reducing the availability of valuable training trajectories. Thus, the richer harness serves a concrete purpose: it enables successful reasoning on difficult problems that a simpler generation process struggles to solve.

  2. Reference-conditioned reconstruction requires complete solutions and does not necessarily produce better training signals. A core strength of HAIR is that policy improvement requires no reference information and operates in exactly the same setting as standard outcome-rewarded RL. HAIR improves the current policy through harness-generated experience, without relying on reference answers or complete proofs as additional supervision. This is central to its role as a stronger policy improvement operator for self-improvement. In contrast, constructing training data from clean proofs or asking the student to reconstruct reasoning from correct solutions introduces reference information beyond this setting.

We nevertheless evaluated reference-conditioned distillation on difficult queries with available ground truth using HDPO [1], which conducts hybrid GRPO + privileged self-distillation.

MethodIMO-Proof pass@1IMO-Proof pass@4UGD pass@1UGD pass@4QE pass@1QE pass@4PhD-Paper pass@1PhD-Paper pass@4
35B baseline19.030.845.565.523.347.813.220.2
GRPO38.648.368.883.046.968.920.729.8
HDPO30.245.267.181.140.356.618.926.9
HAIR42.553.369.685.055.076.724.336.5

HDPO underperforms GRPO across all four benchmarks, whereas HAIR consistently improves both pass@1 and pass@4. These results demonstrate that reference-conditioned distillation does not reproduce the benefits of harness-generated training data in our setting.

Recent studies provide further evidence that correct reference solutions do not automatically yield effective reasoning supervision. Kaur et al. [2] find that privileged-context self-distillation can degrade thinking models by suppressing reasoning branches and self-correction. Kim et al. [3] report that fine-tuning on correct, solution-guided trajectories can substantially degrade mathematical reasoning, and associate self-distillation failures with suppressed uncertainty expression and poorer generalization.

The additional generation cost therefore buys access to successful trajectories on challenging problems without requiring complete reference proofs, and our ablation and HDPO comparison demonstrate the downstream value of this investment.

[1] HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillation.

[2] Kaur et al. Rethinking On-Policy Self-Distillation for Thinking Models. 2026.

[3] Kim et al. Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs? COLM 2026.

AuthorsOpenReview ↗

3. Relationship to agentic distillation methods

Thank you for highlighting these related works. We will add these works to the next version of our draft. HAIR shares the broad motivation of transferring the benefits of structured inference into model parameters, but differs in its learning objective, source of improvement, and evaluation regime.

  1. Self-improvement without a stronger solution-generating teacher. HAIR uses the current policy itself within the harness to discover solutions beyond its native rollout capability, then absorbs these improvements through alternating SFT–GRPO training. As the policy improves, it also strengthens subsequent harness-based generation, forming a closed self-improvement loop. In contrast, AgentArk’s main experiments distill larger teachers, such as Qwen3-32B, into smaller students; Back-Translated Traces uses GPT-4.1-mini for solution generation and back-translation to train Qwen2.5-Math-7B; and Chain-of-Agents obtains its initial supervision from an external OAgents system. HAIR therefore investigates a distinct question: whether improvements induced by an external operator acting on the current policy can be internalized back into that same policy, rather than transferring solutions from a stronger teacher.

  2. Advanced mathematical tasks and substantial gains at large model scales. HAIR demonstrates effectiveness on competition mathematics, undergraduate mathematics, doctoral qualifying examinations, and research-level problems. We also demonstrate improvements at both 35B and 397B, with the 397B model achieving competitive performance against leading open-source models. These results demonstrate HAIR’s effectiveness at tasks and scales relevant to frontier LLM training.

  3. Internalizing harness improvements into standalone reasoning. HAIR improves the backbone’s direct, harness-free reasoning ability: the trained model solves problems without invoking the generation-time harness. This differs from Chain-of-Agents [3], which trains a unified agent policy that continues to interact with external tools during inference.

[1]Luo, Yinyi, et al. "Agentark: Distilling multi-agent intelligence into a single llm agent." arXiv preprint arXiv:2602.03955 (2026).

[2]Huang, Xingyue, et al. "Distilling Tool Knowledge into Language Models via Back-Translated Traces." arXiv preprint arXiv:2506.19171 (2025).

[3]Li, Weizhen, et al. "Chain-of-agents: End-to-end agent foundation models via multi-agent distillation and agentic rl." arXiv preprint arXiv:2508.13167 (2025).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.