acceptodds
Under review as a conference paper at ICLR 2027

InScribe: Internalizing Scaffolds through Agentic Reinforcement Learning

Abstract

Reinforcement learning with verifiable rewards trains coding and terminal agents from expensive rollouts, yet a task on which every rollout fails yields no group-relative solver signal and is discarded. Such failures need not mark the model's limits: with the weights frozen, different harnesses—the programs that organize the agent's planning, tool use, and verification—recover a substantial share of them, and a short teacher diagnosis of the failed attempts recovers more of the remainder. Can these gains outlast the support that made them possible? We treat harnesses and hints as scaffolds: supports put up during training and taken down once the policy has learned what they made reachable. InScribe is a two-stage on-policy RL recipe for internalizing the benefits of both. In Stage I the policy writes its own harness before each task, so that choosing how to work becomes one of its actions; each harness is used once and discarded. In Stage II, training continues on the tasks that a Stage-I probe leaves unsolved under both the stock harness and sampled rewrites, keeping harness generation and adding a verifier-informed teacher diagnosis as context rather than as a prediction target. Main evaluations run under fixed harnesses, with no generated harness and no teacher hint. A 9B student trained this way gains 10.1 points on Terminal-Bench 2.0 and 11.0 on SWE-bench Verified over its base model; the gains hold under harnesses it never trained with and transfer to function-calling and interactive-agent benchmarks it never saw. What the scaffolds taught lives in the weights.

Primary area:
reinforcement learning
Keywords:
agentic reinforcement learning, large language models, harness evolution, tool-using agents

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_PcyKOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes InScribe, a two-stage reinforcement learning recipe for improving coding and terminal agents using training-time scaffolds. In Stage I, the policy generates a task-conditioned harness specification and solves the task under that harness, with separate loss channels for harness generation and task execution. Stage II continues training on tasks that a frozen Stage-I probe fails to solve, mixing clean records with records containing fixed teacher diagnoses informed by failed trajectories and private training verifiers. On Qwen3.5-9B, the full method improves Terminal-Bench 2.0 and SWE-bench Verified by 7.49 and 6.74 percentage points over the authors' 300-update stock-harness RL baseline, with additional transfer results on function-calling and interactive-agent benchmarks.

Critical strengths
  1. The paper highlights an important aspect of agent RL: failures can arise from limitations of the execution harness as well as the policy, yet harness optimization has received comparatively less attention during RL training. By allowing the policy to adapt its own harness, InScribe makes execution strategies part of the learning process. Combining this adaptation with temporary diagnostic guidance provides a useful way to improve the policy.

  2. The reported improvements are promising and extend beyond a single evaluation harness. Table 3 shows consistent checkpoint ordering under three fixed harnesses, including two absent from training. The broader evaluations also suggest transfer beyond terminal and repository-repair tasks. These results support the claim that the trained weights improve, even though they do not by themselves identify what was internalized. The ablations of Table 4 are reasonable and the results also validate the effectiveness of InScribe recipe.

  3. The paper is well writen and easy to follow.

Critical weaknesses
  1. The evaluation is limited to a single dense model, Qwen3.5-9B. Although the results demonstrate transfer across tasks and evaluation harnesses, generalization across model sizes and architectures remains untested. Experiments with larger models and alternative architectures, such as mixture-of-experts models, would help establish whether the benefits of InScribe extend beyond the current student model.

  2. The baseline comparisons could be more comprehensive. The experiments compare InScribe with stock-harness RL and several ablations, but do not include the other harness-evolution methods discussed in the introduction and related work. Comparisons with representative methods under a common student model and task setting would help establish InScribe's advantages over existing approaches.

Questions for authors
  1. Section 4.1 states that the main evaluations omit harness generation, would allowing InScribe to generate harnesses at inference, as it does during training, can further improve performance across the benchmarks?

  2. Table 3 shows that the generated harnesses still underperform the hand-designed Terminus2 and OpenHands harnesses. To what extent does the fixed schema and runtime inherited from the stock harness constrain the scope of adaptation? Could a more flexible harness-generation strategy close this gap or outperform a carefully engineered harness?

  3. As an exploratory question, does the policy exhibit any distinctive behaviors at inference after InScribe training? For example, when deployed under a fixed harness without an explicit harness-generation step, does it spontaneously construct or adapt tools to the task? Qualitative examples would be interesting for understanding what the policy has internalized.

4 comments
AuthorsOpenReview ↗

Response to Reviewer PcyK

Thank you for your careful review and for asking about generalization and what the policy can do under a fixed harness.

W1

We focused our experiments on the dense Qwen3.5-9B student and limit our empirical claims to that model given the time and computing resources required for long-horizon RL. We are keen to evaluate InScribe on larger mixture-of-experts (MoE) models. With harness generation disabled and teacher diagnoses removed at evaluation, InScribe still improves performance on Terminal-Bench 2.0 and SWE-bench Verified under fixed evaluation harnesses (Tables 2 and 3). In future work, we plan to explore training InScribe on larger MoE models and test whether these gains carry over.

W2

In Table 4, InScribe and Stock-harness RL start from the same Qwen3.5-9B initialization, receive 300 optimizer updates each, and are evaluated under the same fixed benchmark harnesses. Relative to this control, InScribe improves Terminal-Bench 2.0 by 7.49 percentage points and SWE-bench Verified by 6.74 percentage points.

We cite HASE and EvoHarness-RL in Section 2 for their related harness adaptation mechanisms. HASE was evaluated on classification and program search, and EvoHarness-RL on household and web interaction tasks. A head-to-head comparison on our terminal and repository tasks would require adapting both methods and running additional long-horizon RL training with the same student, task set, and training budget. For context, one full InScribe training run used 5,695 A100 GPU-hours and launched 64,121 CPU sandbox episodes (Table 9). We would like to explore adapting both methods to our environments for such a comparison in future work.

Q1

We report the inference-time harness generation results in §4.3 and Table 7 of Appendix B.1. With the Stock harness, InScribe scored 54.47 ± 0.70%. Generating a task-specific harness at inference increased the score to 57.27 ± 0.70%, a gain of 2.80 percentage points. Together with the 3.37-percentage-point gain on Terminal-Bench 2.0, this shows that inference-time harness generation improves performance over the Stock harness on both of our primary benchmarks.

Q2

Our generated harnesses use the configuration format and runtime of the Stock harness. For each task, the model can add executable Python tools and adjust the tool-use rules and history retention. As detailed in Appendix C.1 (Table 11), verification and recovery rules are rendered as instructions for the model. For example, a generated configuration can ask the model to run tests after a change. The model still has to invoke a tool to run them. This limits which execution behaviors a generated harness can control directly.

Table 3 compares the same model across complete harnesses. A controlled comparison would be needed to measure how much the fixed configuration and runtime contribute to the gap. Giving the model a choice of mechanisms that the runtime can execute for each task, such as context summarization for long tasks or automatic self-checks before submission, could help narrow the gap. Whether this approach can match or surpass a hand-designed harness remains to be tested with the same checkpoint, tasks, verifier, and total inference budget.

Q3

We observed the policy construct a task-specific tool in a pilot TB2 evaluation of an earlier Stage II checkpoint. The evaluation used a fixed Terminus2 harness with harness generation and teacher hints disabled at inference. In cancel-async-tasks, the policy used the existing terminal interface to build and run a task-specific tool that checked concurrency and cleanup callbacks. After the tool reported an empty cleanup list, the policy revised the implementation and its checks, reran the tool, and passed the benchmark verifier. We describe this case in Appendix C.4 (A task-specific check under a fixed harness) and refer to it in Section 4.3. How often this behavior occurs across different checkpoints remains to be studied.

Thanks for the careful explaination. It is a pity that we can not have detailed generalization experiments results and more methods comparison due to heavy training budget.

Accord to the results in section 4.3 and Table 7 of Appendix B.1, although InScribe do help improve the final performance compared to naive Stock-harness RL (16.85 -> 19.85 vs 16.85 -> 17.23), there is still large gap between stock harness and generated harness ( 19.85 vs 23.22). In other words, there is still large room for the "Internalizing".

Another question about the transfer across differencr harness. As shown in Table 3, why training in a stock based harness would improve most in the Terminus 2 harness? Would it be related to the capability of the base harness?

Thank you for the follow-up. With Stock fixed, InScribe scores 19.85% compared with 17.23% for Stock-harness RL. Enabling harness generation raises the same InScribe checkpoint’s score to 23.22%, while it lowers the Stock-harness RL checkpoint’s score by 1.13 percentage points. Writing a useful harness for the task and using it effectively are themselves capabilities the model learns through training.

We believe the harness plays different roles in training and evaluation. On TB2, the base model scores 20.97% with Terminus2 and 16.85% with Stock. With the same Stage I recipe, the checkpoint trained from Stock Bash scores 26.22% under fixed Terminus2, compared with 21.35% for the checkpoint trained from Terminus2 itself (Table 3). We believe Stock Bash’s simpler interaction protocol may provide an easier starting point. Through repeated practice writing and executing task-specific harnesses, the model can gradually learn strategies that help it use more demanding interfaces such as Terminus2 more effectively. This may help explain why the gains are larger under Terminus2.

Thanks for the further feedback. I have no further concerns and will maintain my score of 3 for acceptance.

Reviewer_yBbiOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

InScribe has two stages of on-policy RL. In Stage I, the policy writes a JSON harness before each task and solves the task under it. In Stage II, training continues on the 2,500 tasks that a probe still fails. Each task is paired with a short diagnosis from DeepSeek-V4-Flash, which sees the failed traces and the private verifier. Qwen3.5-9B goes from 20.97 to 31.09 on Terminal-Bench 2.0 and from 49.67 to 60.67 on SWE-bench Verified. It beats TMax-9B and a 300-update stock-harness RL run, and the gains carry over to BFCL and τ²-Bench.

Critical strengths
  • Strong 9B results, including on benchmarks the recipe never trained on (Table 2). The ranking of checkpoints holds under two harnesses never seen in training (Table 3).
  • The engineering is well documented (harness schema, teacher prompt, run footprint).
Critical weaknesses
  1. The claim of model internalizing scaffolds is not supported. The title and abstract say the scaffolds are internalized and that what they taught "lives in the weights" (l. 029, 475). If the model had taken over what a harness does, it should need a good harness less after training. Under the stock harness used in training, InScribe beats Stage I by only 1.13 points (one TB2 task). Its 19.85 is below the base model under Terminus2 (20.97). So the claim the data support is that training makes the model better at writing harnesses and at working inside a good one. Please reframe the claim this way, or show that the gap between weak and strong harnesses shrinks.

  2. Stage II relies on a stronger teacher that reads the hidden tests, and the paper does not show how much of the gain comes from this. The teacher sees the private verifier script. Its hints "name source files, faulty conditions, and repair strategies" (l. 1205), and the Pillow hint is close to a fix (l. 1279–1280). This is the setting of RL with hints that use privileged information or a stronger model, as in LEAP (ICLR 2025), NuRL, QuestA and Scaf-GRPO (ICLR 2026), and LUFFY (NeurIPS 2025). None of these is discussed.

Questions for authors
  1. What do the base model and TMax-9B score under the stock harness (Table 3)?
  2. What does the teacher itself score on TB2 and SWE-V? How does Stage II compare with hints from a teacher without the verifier?
  3. How much of the harness change is just prompt text? If you keep only the prompt changes, or only the runtime changes (tool policy, the extra confirmation turn, Python tools), how much of the Generated-column gain in Table 3 remains?
3 comments
AuthorsOpenReview ↗

Response to Reviewer yBbi (W1–W2 and Q1)

Thank you for your careful review and for suggesting the verifier-access and harness-component comparisons.

W1

Our original wording about internalization was unclear. By internalization, we mean that as the policy learns to write harnesses, it also learns to use them more effectively, for example by using tool feedback to choose its next action. The results in Table 3 support your observation that a strong harness remains valuable after training.

As you note, under fixed Stock Bash, full InScribe improves on Stage I by a modest 1.13 percentage points and still scores below the base model under Terminus2. To test whether the learning carries over, we compare checkpoints under the same fixed harness with generation and teacher hints disabled. Full InScribe outperforms Stock RL by 2.62 points under Stock Bash, 5.62 under OpenHands, and 7.49 under Terminus2. The gain from switching from Stock Bash to Terminus2 grows from 6.37 points for Stock RL to 11.24 for InScribe, consistent with the trained policy making better use of a strong harness.

We clarify this meaning of internalization in the revised abstract, §4.3, and conclusion.

W2

In the matched Stage II comparison, removing the private training verifier from the teacher’s inputs lowered the student’s mean score by 0.75 percentage points on Terminal-Bench 2.0 (TB2) and 1.00 point on SWE-bench Verified. Both means remained above the no-diagnosis control (Table 5). The teacher still saw the task description and the student’s failed attempts. The same verifier provided the student’s RL reward, and the student was evaluated without hints.

We agree that the verifier is privileged information and that the Pillow example provides concrete repair guidance. With verifier-informed diagnoses, the student scored 5.25 points higher on TB2 and 3.27 points higher on SWE-bench Verified than in the matched no-diagnosis continuation (Table 4).

We now discuss LEAP’s privileged corrections, the hints or partial solutions used by NuRL, Scaf-GRPO, and QuestA, and LUFFY’s off-policy demonstrations in §2. InScribe applies diagnoses to tasks that remain unsolved after Stage I harness exploration. The policy learns from its own rollouts on paired versions of each task with and without a diagnosis. We clarify the teacher’s inputs in §3.3 and mark the corresponding revisions in blue.

Q1

We evaluated the Qwen3.5-9B starting checkpoint and the released TMax-9B checkpoint on Terminal-Bench 2.0 using the same fixed Stock Bash harness as in Table 3. For reference, we also include the OpenHands results and the Terminus2 scores reported in Table 2. Scores are mean ± sample standard deviation over three runs.

CheckpointStock Bash (%)Terminus2 (%)OpenHands (%)
Qwen3.5-9B16.85 ± 1.1220.97 ± 1.7219.10 ± 1.12
TMax-9B17.98 ± 1.1322.85 ± 1.3021.35 ± 1.12

Under the same Stock Bash harness, InScribe scores 19.85 ± 1.71%, with a mean 1.87 percentage points higher than the released TMax-9B checkpoint.

AuthorsOpenReview ↗

Response to Reviewer yBbi (Q2–Q3)

Q2

We evaluated our self-hosted DeepSeek-V4-Flash using the same benchmark harnesses as in the paper, with Terminus2 on Terminal-Bench 2.0 and OpenHands on the full SWE-bench Verified set. Scores are reported as mean ± standard deviation.

Teacher configurationTB2 (%)SWE-bench Verified (%)
Thinking disabled42.70 ± 1.1267.07 ± 0.70
Thinking enabled53.93 ± 1.1273.13 ± 0.81

The two Stage II runs use the same training configuration and differ only in whether the teacher can read the private training verifier when generating hints. In both runs, the teacher receives the task description and the student’s failed attempts, and the same task verifier provides the student’s RL reward. We evaluated both student checkpoints under the same benchmark protocol. At evaluation, the students solved tasks independently without teacher hints or access to verifier scripts. The benchmark verifiers were used only for scoring.

Stage II settingTB2 (%)SWE-bench Verified (%)
Teacher with verifier access (Table 4)31.09 ± 1.2960.67 ± 1.72
Teacher without verifier access (matched run)30.34 ± 1.1259.67 ± 0.70

Removing the verifier from the teacher’s inputs lowers the observed mean by 0.75 percentage points on TB2 and 1.00 point on SWE-bench Verified in this matched comparison. These results suggest that the task description and observed failures provide useful information for generating Stage II hints.

Q3

Across three full TB2 evaluations with Generated harnesses, the policy modified the harness in 206 of 267 rollouts (77.2%). Of these edits, 137 (66.5%) changed only the solver prompt, and 69 (33.5%) changed both the prompt and runtime settings.

HarnessTB2 (%, mean ± SD)Gain over Stock (pp)
Stock (Table 3)19.85 ± 1.71—
Prompt-only21.35 ± 1.13+1.50
Runtime-only20.22 ± 1.13+0.37
Full Generated (Table 3)23.22 ± 1.30+3.37

Using the Stock score in Table 3 as the reference, Prompt-only retains about 45% of the observed mean gain of the full Generated setting, and Runtime-only retains about 11%.

The complete harness has the highest mean score in this comparison. The prompt guides what the solver plans and checks, and the runtime governs the tools and execution rules it uses to carry out that plan. Stage I lets the policy adapt these two parts together and solve the task with the configuration it chose. The verifier outcome gives the policy feedback on how well those choices work together.

In Table 3, the gain from each checkpoint’s own generated harness over its fixed Stock score rises from −1.13 points for Stock RL to +1.50 for Stage I and +3.37 for full InScribe. Together with the fixed-harness gains, this suggests that the policy develops a better practical understanding of how to configure the prompt and runtime together and use the resulting harness to solve tasks. We will report the breakdown and ablation in the revised manuscript.

Thanks you for the explanation. I will keep my score.

Reviewer_BWwwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper puts forward InScribe, an RL recipe for internalizing the knowledge and performance gains that harnesses and hints provide. More specifically, the paper first shows that changing the harness of a model (from a default one) or providing model with verifier-informed hints can help it solve tasks that it was not able to solve (and hence learn during RL training). The paper then provides an RL recipe for internalizing this into the model weights. During RL training, in stage 1 of the recipe, the model writes a harness for each task it's trying to solve and then attempts the task and receives credit for the harness and the task separately. In stage 2, the tasks that still fail receive a hint from a teacher that has access to the private verifier and RL training continues again (with harnesses being written for each task.) The paper shows their method helps models provide some gains on agentic and function calling tasks with generalization to harnesses not seen during training.

Critical strengths
  • The topic of the paper is quite interesting. There has been a number of works in the last year which have tried to internalize hints provided during RL training (e.g., self-distillation methods). But this paper goes a step further, and asks whether we can also internalize some of the benefits that harnesses provide.
  • In particular, I hadn't seen any work letting agents write their own harnesses, and I really liked this idea.
  • The gains are relatively significant and the authors have shown the gains transfer to other harnesses and tasks. Yet, there are some concerns regarding this, see below.
Critical weaknesses
  • There's still relatively little explanation why stage 1 helps the model once it's bound to a fixed harness.
  • Experiments could have been more substantial. Currently the paper has only one training set, one model, and one seed. So each method and baseline/ablation is just based on one observation. Having multiple seeds, multiple models and datasets would have been nice.
  • I think the paper can be more upfront about the limitations and its results, for example the 10% gain in the abstract it computed with respect to the base model not a baseline. (The point above makes this even more important.)
  • See the first question below as well.
Questions for authors
  • Q1. What is very surprising to me is that Table 2 results are reported using Terminus2 harness and not the Stock harness the model were trained with. What would the results look like if they were reported under the training harness. Or similarly, what would the results be if Terminus2 was used at training? A hypothesis is that the model was just bad at using Terminus2 and learns to use it at test time as it generates and interacts with harnesses. What if this is all about learning to use the harness well. How can the small gains on the Stock harness be explained?
  • Q2. In Figure 1 (harness swap experiment), if the same harness was repeated 8x2=16 extra times it might have recovered some of the tasks. How should we measure that? Can you provide that number (i.e., 16 extra times for all the tasks, how many get solved)?
  • Q3. How can we compare stage 2 with other methods for internalizing hints such as self-distillation?
  • Q4. Why harness generation transfers to cases where harness is not generated? What is being internalized here?
  • Q5. In stage 1, six harnesses are generated based on predefined directions and two are generated freely. Hence they have different prompts. If that's the case, how can we justify putting them in one GRPO group? Similarly, the AisolA_i^{sol} group are coming from different harnesses, should they be in the same GRPO group?
  • Q6. The paper says "Stage I lifts the training distribution; Stage II lifts the rest."; can you please elaborate why that's the case?
  • Q7. Table 2 does not have the harness generation right? So in line 370, "there is no generated harness" is also already the case in Table 2, right?
5 comments
AuthorsOpenReview ↗

Response to Reviewer BWww (W1–W4 and Q1)

Thank you for reading our paper so carefully and for proposing concrete comparisons. Your questions helped us identify where the training and evaluation settings needed clearer explanation. We will incorporate the changes described below in the revised manuscript and mark the corresponding passages in blue for ease of reference.

W1

During training, harness rewrites let the solver try different workflows for the same task. When some rollouts pass and others fail, the task reward gives the solver a signal for the actions taken in the successful rollouts. In the cron example (Appendix C.2), the successful rollout installs the job before saving and checking the crontab, while the paired Stock rollout saves it before installation. The successful sequence uses Bash commands already available under Stock, so decisions about when to act and when to check the result can carry over. In Table 3, Stage I improves the mean score over Stock RL under all three fixed harnesses, including Stock Bash (18.72% vs. 17.23% on TB2). We will expand §4.3 to explain this connection between the training rollouts and the fixed-harness results.

W2

We have yet to test whether the training results generalize across seeds, base models, and datasets. Online rollouts make agentic RL expensive to repeat, and related studies have also reported comparisons under fixed training configurations, including Self-Play SWE-RL (ICML 2026), MobileRL (ICLR 2026), CodeGym (ICLR 2026), and DAPO (NeurIPS 2025). For our terminal-based coding tasks, one complete two-stage InScribe run used 5,695 A100 GPU-hours and recorded 64,121 sandbox episode launches (Table 9). We evaluated the same checkpoint three times on Terminal-Bench 2.0 and SWE-bench Verified, using 480 GPU-hours and 1,767 sandbox calls in total. We report this evaluation cost in Appendix B.3.

W3

In §4.4 (Table 4), InScribe and Stock RL each receive 300 optimizer updates and use the same fixed evaluation harness for each benchmark. InScribe scores 7.49 percentage points higher on Terminal-Bench 2.0 and 6.74 points higher on SWE-bench Verified. Relative to the Qwen3.5-9B base model, the gains are 10.12 and 11.00 points, respectively (Table 2).

W4

We address the training and evaluation comparisons in Q1 and will make the distinction between these settings explicit in §4.3.

Q1

On Terminal-Bench 2.0 (TB2) under fixed Stock Bash, Stage I scores 1.49 percentage points above Stock RL and full InScribe scores 2.62 points above it. When we reran Stage I with Terminus2 as the starting harness, that checkpoint scored lower than the Stock-started checkpoint under all three fixed evaluation harnesses (Table 3).

Table 2 uses fixed Terminus2 for every model to test transfer to an interface held out from InScribe’s original training. We started training from Stock Bash following TMax’s choice of a simpler tool interface for RL (Ivison et al., 2026, Appendix D). The two Stage I runs use the same setup apart from the starting harness. The table below shows results under Stock Bash and Terminus2 as means ± sample standard deviations over three evaluation runs. Table 3 also includes OpenHands.

Training conditionStock Bash evaluation (%)Terminus2 evaluation (%)
Stock-harness RL17.23 ± 1.3023.60 ± 1.13
InScribe Stage I, starting from Stock Bash18.72 ± 1.7226.22 ± 1.30
InScribe Stage I, starting from Terminus216.85 ± 1.1221.35 ± 1.12
Full InScribe19.85 ± 1.7131.09 ± 1.29

Holding the evaluation harness fixed, full InScribe improves on Stock RL by more under Terminus2 (7.49 points) and OpenHands (5.62 points) than under Stock Bash. Holding the checkpoint fixed and switching the evaluation harness from Stock Bash to Terminus2 adds 6.37 points for Stock RL and 11.24 points for full InScribe. This pattern is consistent with the trained policy making better use of a stronger harness (§4.3).

AuthorsOpenReview ↗

Response to Reviewer BWww (Q2–Q5)

Q2

Using the same frozen checkpoint and Stock Bash harness, we ran 16 additional rollouts per task on the 100 tasks in Figure 1. We counted a task as recovered if any additional rollout passed its verifier. Four tasks were recovered, each with exactly one successful rollout under the default setup.

In the harness-swap comparison, we ran each of the same 100 tasks eight times with Terminus2 and eight times with OpenHands v2. Across the two sets of rollouts, 32 distinct tasks were recovered. This gives 16 additional attempts per task in the harness-swap comparison, matching the number of additional Stock retries. The initial eight Stock attempts were used to select the 100 tasks.

Q3

To assess a related alternative to self-distillation, we ran on-policy distillation (OPD) with Qwen3.5-27B as the teacher. A stronger 9B agentic coding checkpoint was unavailable in our setup. For a fair comparison with InScribe Stage II, both methods start from the same 9B Stage I checkpoint and are evaluated under the same fixed harnesses without hints. The OPD teacher provides token-level supervision on the student’s own rollouts. In InScribe Stage II, the diagnosis guides exploration and the verifier supplies the reward for the student’s attempts.

MethodTerminal-Bench 2.0 (%)SWE-bench Verified (%)
Stage I only (Table 4)26.22 ± 1.3058.47 ± 1.70
Stage II OPD baseline28.09± 1.1258.73± 0.81
InScribe Stage II (Table 4)31.09 ± 1.2960.67 ± 1.72

In this comparison, InScribe Stage II scores 3.00 percentage points higher on TB2 and 1.94 points higher on SWE-bench Verified. We will add the OPD comparison to §4.4 and state the teacher, student initialization, and evaluation protocol alongside it.

Q4

In Stage I, the generated harness shapes the workflow, and the task reward provides a learning signal for the policy’s solving actions. In the Go/C example (Appendix C.2), both training rollouts encounter missing C headers. The Stock rollout switches to pure Go. With CGo configuration made explicit in the generated harness, the policy inspects the include directory, installs the missing development package, and completes the required CGo build.

The case illustrates how the policy practices using tool feedback to recover from an error while preserving the task’s requirements. A fixed Bash harness still provides the shell tools and compiler output needed for that kind of decision. With generation disabled, Stage I’s mean scores exceed those of Stock RL by 1.49, 2.62, and 3.57 percentage points under Stock, Terminus2, and OpenHands, respectively (Table 3). In §4.3, we use internalization to mean that the policy learns to use harnesses more effectively as it learns to write them. The gains under fixed harnesses with harness generation and teacher hints disabled provide evidence that this ability carries over to evaluation.

Q5

In Stage I, we group rollouts by task. The six directed slots and two free slots are assigned before sampling to explore different harness rewrites for the same task, with a shared stock anchor for scoring those rewrites. During solving, each candidate is conditioned on the harness actually executed in its rollout. All eight candidates are scored by the same verifier, so their solver rewards are comparable across harnesses. We center harness scores and solver rewards separately over the eight candidates and exclude the stock anchor from both means.

For either score sᵢ, the mean centering in Eq. (3) gives sᵢ − s̄ = (7/8)(sᵢ − s̄₋ᵢ), where s̄₋ᵢ is the mean of the other seven scores. With independent rollout draws conditional on the task, stock anchor, and sampling policy, the mean of the other seven scores is independent of the sampled actions of candidate i, even when the candidates have different prompts or harnesses. In Eq. (4), each token’s policy ratio uses its actual sampling context, including its generation prompt or executed harness. We will clarify the task-level baseline and the context used for each token’s policy ratio in §3.2.

AuthorsOpenReview ↗

Response to Reviewer BWww (Q6–Q7)

Q6

Our earlier description of where Stage I and Stage II improve performance was too broad. In Stage I, rewriting the harness lets the policy try different workflows for the same task. When some rollouts succeed, the policy can learn which actions work and reuse them on tasks with similar workflows. In Table 2, Stage I improves on terminal and code repair benchmarks that resemble the training tasks. It also outperforms the base model on BFCL-v3.

Stage II focuses on tasks for which every rollout in the Stage I probe failed. A diagnostic hint can prompt the policy to revisit overlooked evidence or a mistaken assumption, helping it find a solution that passes the verifier. In the SQLite example in Appendix D.4, the hint directs the policy to check its output against the raw order records, independently of the query it is repairing. Stage II also trains on unhinted versions of the same tasks, giving the policy opportunities to apply these practices without guidance. In the matched comparison in Table 4, both conditions are evaluated without hints. Adding diagnostic hints during training improves TB2 and SWE-bench Verified by 5.25 and 3.27 percentage points, respectively.

In Table 2, Stage II improves 9 of the 10 scores relative to Stage I. Practices such as independent verification also apply to other tool-use tasks, offering one possible explanation for the broader gains observed after Stage II.

Q7

All evaluations in Table 2 use fixed benchmark harnesses without harness generation or teacher hints. That is the same no-generation setting described in line 370. The Generated column in Table 3 is the separate inference setting in which the policy writes a harness for each rollout, also without hints. We will make this distinction explicit in the Table 2 caption and §4.3.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Discussion

We are grateful to the area chair and reviewers for the care they took in reviewing InScribe and the time they spent discussing it with us. We especially appreciated their questions about harness choice, training baselines, and diagnostic hints. Those questions helped us refine the evaluation and explain more clearly what the policy learns during training.

What we added. We expanded the fixed-harness evaluations and compared two starting harnesses for Stage I. We also measured how many tasks Stock Bash recovers with the same number of attempts as switching harnesses, and tested the gains from generating a harness at inference and from changing its individual components (Table 3 and Appendices A and B.1).

For Stage II, we evaluated the teacher on its own and measured how the student’s results change when the teacher generates hints without access to the training verifier. We also added an on-policy distillation baseline with a Qwen3.5-27B teacher (Table 5 and Appendix D.5).

Together with the original results, these comparisons give us a clearer picture of what the policy learns to do with a harness and how diagnostic hints help it learn from tasks that remain unsolved after Stage I.

What carries over from Stage I. In one training example, a CGo build failed because C headers were missing. The generated harness made the CGo configuration explicit. The policy inspected the include directory, installed the missing development package, and completed the required build. The shell tools and compiler feedback needed for those steps are also available under fixed Bash (Appendix C.2). By internalization, we mean that as the policy learns to write harnesses, it also learns to use them more effectively.

To test whether this ability carries over, we compare checkpoints under the same fixed harness, with harness generation and teacher hints disabled. Table 2 now makes this protocol explicit. The Stage I checkpoint already scores above Stock RL under all three fixed harnesses (Table 3). At 300 optimizer updates for each method, full InScribe improves over Stock RL on Terminal-Bench 2.0 (TB2) by 2.62 percentage points under fixed Stock Bash and 7.49 under fixed Terminus2, and by 6.74 points on SWE-bench Verified (Tables 3 and 4).

What diagnostic hints add in Stage II. On tasks that remain unsolved after the Stage I probe, a diagnostic hint can draw the policy’s attention to an error or overlooked evidence. The verifier checks whether the attempted correction works. In matched Stage II continuations, adding hints improves TB2 by 5.25 points and SWE-bench Verified by 3.27 points (Table 4). The hinted continuation also scores above the no-hint run when the teacher sees only task descriptions and failed traces (Table 5). For a follow-up comparison, we used the same hints as the full Stage II RL run to collect trajectories for offline distillation. Both methods started from the same Stage I checkpoint and were evaluated under the same fixed harnesses with hints disabled. Stage II RL scored 2.63 percentage points higher than offline distillation on TB2 and 4.07 points higher on SWE-bench Verified. See the follow-up comparison.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.