acceptodds
Under review as a conference paper at ICLR 2027

LiteExec: Scaling Down the Executor in Long-Horizon Agents

Abstract

Long-horizon tasks involve different capability requirements, from high-level planning and task decomposition to direct execution and tool interaction. Monolithic agents apply a single model throughout, potentially misallocating model capacity and increasing inference costs. In this work, we investigate whether these differences can be exploited by pairing a large-model planner with a smaller executor. With this role separation, we find that scaling down model size for the executor under a large-model planner leads to a smaller performance drop than scaling down a monolithic agent, suggesting that execution is more amenable to scaling down. Additionally, the planner consumes less than of total tokens, leaving most token usage to smaller executors. However, off-the-shelf executors remain limited in performance and token efficiency due to ineffective collaboration with the planner. To address these limitations, we propose Executor GRPO, a simple yet effective reinforcement learning approach that trains a small executor using end-to-end task rewards with a frozen planner. The resulting model, LiteExec, exhibits emergent collaborative behaviors during training, including more informative feedback and more frequent reporting of execution difficulties. Experimental results demonstrate that pairing large-model planners with LiteExec improves performance and substantially reduces estimated inference costs relative to off-the-shelf executors, with improvements generalizing across planners and from coding to search tasks. Compared with the corresponding monolithic large-model agents, large-model planners paired with LiteExec-9B achieve competitive scores of (vs. ) on SWE-bench Verified and (vs. ) on SWE-bench Multilingual, while reducing large-model token usage by approximately and and estimated total inference costs by and , respectively.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Agents

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_QPniOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies a large planner paired with a smaller executor for coding and search agents. Executor GRPO updates only the executor, using the final task reward shared across its execution rounds while keeping the planner frozen. Training Qwen3.5-9B with a Hy3 planner improves both task scores and token use relative to the untrained executor. For example, SWE-bench Verified rises from 65.6 to 70.0 while executor tokens fall from 6.17M to 1.63M per task. Evaluations with two other planners and two search benchmarks test transfer. An open-data training variant, a monolithic-GRPO comparison, and response analyses provide additional evidence about the learned specialization.

Critical strengths
  1. The improvement over the untrained executor is substantial and appears across planners and domains. The coding-to-search transfer is particularly useful evidence that the adaptation learns something beyond the training task interface.

  2. Table 3 is an informative test of specialization: LiteExec performs worse alone but better under the planner. This supports the practical case for training an executor in its intended interaction setting. The cost analysis is also unusually candid about provider prices and caching; Appendix C reports a setting where the advantage over a monolithic model reverses.

Critical weaknesses
  1. The architectural comparison is also restricted to a monolithic agent and a fixed sequential planner-executor arrangement. Why were other multi-agent variants not considered, such as an orchestrator that dynamically delegates to multiple or specialized workers, with parallel execution where appropriate?

  2. How would ordinary GRPO perform if its reward jointly accounted for task success and inference cost? Would Executor GRPO still offer an advantage under this objective?

Questions for authors

please see Weaknesses 1 and 2

2 comments
AuthorsOpenReview ↗

Response to Reviewer QPni (Part 1)

Thank you for your thoughtful review. We appreciate your recognition of the strengths of our work and your suggestions for improving the paper. We have considered each of your comments and hope our responses below address your concerns.

  • The architectural comparison is also restricted to a monolithic agent and a fixed sequential planner-executor arrangement. Why were other multi-agent variants not considered, such as an orchestrator that dynamically delegates to multiple or specialized workers, with parallel execution where appropriate?

Thank you for raising this point. We agree that evaluating a broader range of agent harnesses would be valuable, and we appreciate the opportunity to clarify why we chose a sequential setup for this study.

Why we chose a simple sequential setup. Our primary research question is whether a small model can serve as an executor and carry out most of the execution work in collaboration with a large model. The fundamental challenge is whether the small model can collaborate effectively, for example, by providing supporting evidence and clearly reporting whether it completed its assigned subtask. These requirements apply across agent harnesses, whether execution is sequential or parallel. We therefore use a simple sequential setup to focus on the executor’s ability to collaborate. Harnesses with multiple or specialized workers raise additional questions about how effectively the planner can assign work, coordinate workers with different capabilities, and resolve conflicts between their actions. These challenges primarily concern the planner’s orchestration capabilities. Given our focus on the executor, we leave a systematic evaluation of these settings for future work.

Evaluation with an additional agent harness. To further assess whether the benefits of executor training extend beyond the sequential setup, we implemented a harness variant that allows the planner to invoke multiple executors within a single round (with the planner prompt explicitly permitting parallel delegation). We compare Qwen3.5-9B and LiteExec-9B under concurrency limits of one (as in the paper), two, and four executors. The planner determines whether and when to delegate subtasks in parallel. We use Hy3 as the planner and the same benchmark tasks across settings, and report the mean and sample standard deviation across three evaluation runs for each setting.

Max. concurrent executorsExecutorSWE-bench Verified (% resolved) ↑SWE-bench Multilingual (% resolved) ↑
1Qwen3.5-9B66.4 ± 1.259.8 ± 1.9
1LiteExec-9B70.6 ± 1.067.0 ± 0.9
2Qwen3.5-9B66.9 ± 1.558.6 ± 1.0
2LiteExec-9B69.4 ± 0.963.0 ± 0.3
4Qwen3.5-9B67.6 ± 2.058.2 ± 0.7
4LiteExec-9B69.5 ± 0.664.2 ± 0.7

Across all evaluated harness settings, LiteExec-9B achieves higher task resolution rates than the off-the-shelf Qwen3.5-9B executor on both benchmarks. These results suggest that the benefits of executor training extend beyond the sequential setup to a harness that supports parallel execution.

AuthorsOpenReview ↗

Response to Reviewer QPni (Part 2)

How would ordinary GRPO perform if its reward jointly accounted for task success and inference cost? Would Executor GRPO still offer an advantage under this objective?

Thank you for suggesting this comparison. We agree that jointly optimizing task success and efficiency is another promising research direction. We appreciate the opportunity to investigate this approach and compare it with our method.

Evaluation with cost-aware GRPO. To investigate this approach, we use ShorterBetter (Yi et al., NeurIPS 2025), a GRPO method that encourages efficiency by jointly rewarding task correctness and penalizing deviations from the shortest successful response length. We train Qwen3.5-9B on the same training data as LiteExec, using α = 2 and β = 0.001 following the hyperparameter guidance in the paper, with eight rollouts per group. The reward is r = αs − β|L − L*|, where s is the task score, L is the trajectory length, and L* is the length of the shortest fully successful rollout in the group, or the mean length of valid rollouts if none fully succeeds. Further details of ShorterBetter can be found in the original paper.

Yi, J., Wang, J., and Li, S. (2025). ShorterBetter: Guiding Reasoning Models to Find Optimal Inference Length for Efficient Reasoning. Advances in Neural Information Processing Systems, 38.

Results and analysis. We evaluate the ShorterBetter GRPO checkpoints at 50 and 100 training steps as executors with Hy3 as the planner. All configurations use the same benchmark tasks and token prices. We report mean ± sample standard deviation across three evaluation runs. Estimated costs follow the calculation protocol in Table 1 of the paper and include both planner and executor calls.

Executor / TrainingSWE-bench Verified (% resolved) ↑Cost (USD/task) ↓SWE-bench Multilingual (% resolved) ↑Cost (USD/task) ↓
Qwen3.5-9B66.4 ± 1.20.511 ± 0.00959.8 ± 1.90.508 ± 0.019
+ ShorterBetter GRPO (50 steps)68.5 ± 1.20.290 ± 0.00660.2 ± 1.30.320 ± 0.010
+ ShorterBetter GRPO (100 steps)67.4 ± 0.90.122 ± 0.00459.7 ± 2.20.142 ± 0.001
+ Executor GRPO (LiteExec-9B)70.6 ± 1.00.160 ± 0.00367.0 ± 0.90.186 ± 0.005

As shown in the table, ShorterBetter substantially reduces estimated costs, with further reductions from 50 to 100 training steps accompanied by lower mean task resolution rates on both benchmarks. Compared with the 50-step ShorterBetter checkpoint, LiteExec-9B achieves higher task resolution rates and lower estimated costs. Compared with the 100-step checkpoint, its performance advantage increases to 3.2 percentage points on SWE-bench Verified and 7.3 percentage points on SWE-bench Multilingual, at additional estimated costs of USD 0.038 and USD 0.044 per task, respectively. Executor GRPO (LiteExec) lies on the Pareto frontier among the evaluated configurations on both benchmarks, illustrating the trade-off between task resolution and cost in the collaborative setting. Although ShorterBetter jointly accounts for task success and efficiency, its objective remains centered on individual task-solving capability, while Executor GRPO (LiteExec) targets collaboration capability, such as the executor reporting execution evidence and clearly communicating blocked status to the planner. We believe that developing effective collaboration capabilities in agents, either with humans or with other agents, is a promising research direction beyond the prevailing emphasis on independent task solving.

Reviewer_uyRTOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper looks at splitting long agent tasks between a large planner and a small executor. It shows that the small executor does better when trained for this role with reinforcement learning, even when the planner stays fixed. The trained executor improves coding and search results and gets close to using the large model throughout, with much less large-model usage

Critical strengths

The setup is practical, and the experiments tell a clear story. The control against standard training suggests the gain comes from better collaboration, not just a stronger small model. It also transfers to other planners and to search tasks despite being trained on coding only

Critical weaknesses

The cost savings depend on pricing assumptions. As the appendix shows, they shrink or disappear for some planners once caching is counted, so the claims should be stated a bit more carefully. Results also come from single runs, with some search gains being small, and it isn't clear how well this carries over to other planners or agent setups beyond those tested

Questions for authors
  1. Could you bring the caching-adjusted cost comparison into the main text?
  2. Do you have repeated runs, or any sense of the noise on the smaller improvements?
5 comments
AuthorsOpenReview ↗

Response to Reviewer uyRT (Part 1)

Thank you for your careful review and constructive feedback. We appreciate your recognition of the strengths of our work. We have considered each of your comments and hope our point-by-point responses below address your concerns.

  • The cost savings depend on pricing assumptions. As the appendix shows, they shrink or disappear for some planners once caching is counted, so the claims should be stated a bit more carefully.
  • Could you bring the caching-adjusted cost comparison into the main text?

Thank you for raising these points. We agree that the estimated cost savings depend on pricing and caching assumptions, and we appreciate the opportunity to clarify how these assumptions affect the cost comparison and to show how we will revise the main text.

First, API prices reflect several factors beyond model size, including serving infrastructure and provider pricing policies. For example, although Qwen3.5-35B-A3B has fewer parameters than Hy3-295B-A21B, its listed price per million output tokens was higher (USD 0.80 vs. USD 0.528). Pricing differences also extend to cached input. The cached-input rate was about 3% of the standard input rate for DeepSeek V4 Flash, compared with 25% for Hy3. The provider did not list a separate cached-input rate for Qwen3.5, so we used the same 25% ratio as for Hy3 to estimate caching-adjusted costs for Qwen3.5 models and LiteExec. For each planner, we apply the same token rates to LiteExec and the off-the-shelf Qwen3.5-9B executor. This makes the cost comparison more direct because the estimated difference reflects how the two configurations use tokens at the same rates. Under idealized caching, where all eligible input tokens receive the cached-input rate, LiteExec has lower estimated costs than the off-the-shelf executor across all three planners on both coding benchmarks. The cost comparison with the corresponding monolithic agents varies by planner and depends on relative API prices and caching discounts (e.g., the larger discount for DeepSeek V4 Flash). In Appendix C (“Sensitivity to Pricing”), we use the measured token usage in the plan–execution and monolithic configurations to derive the break-even ratio of the effective per-token price for the large model to that for the small executor model, at which the two configurations have the same estimated cost. The break-even ratios range from approximately 0.91 to 1.19 across the evaluated coding settings. These thresholds are close to one, indicating that cost savings do not require a large per-token price gap between the large model and the small executor model.

Second, we will include the caching-adjusted comparison in the main text. We agree that presenting the estimates with and without caching discounts together will help readers see how caching affects the cost comparison with monolithic agents. We have drafted the paragraph below and will add it to Section 4.3 in the next version of the manuscript, immediately after the existing cost comparison without caching discounts. We will also revise the existing cost claims in Section 4.3 to clarify that the estimated savings over monolithic agents depend on the pricing and caching assumptions.

Prompt caching also influences the estimated cost savings. To further investigate this effect, we provide a cost analysis under an idealized caching assumption, where all eligible input tokens receive the cached-input rate (Table 5). Under this assumption, large-model planners paired with LiteExec-9B incur approximately 10%–61% lower estimated costs per task than those paired with the off-the-shelf Qwen3.5-9B executor across all three planners and both coding benchmarks. Compared with the corresponding monolithic agents, these configurations retain a cost advantage with Hy3 and Qwen3.8-27B but incur higher estimated costs with DeepSeek V4 Flash on both benchmarks. This difference reflects the larger cache discount for DeepSeek V4 Flash, whose cached-input rate is 3.18% of its standard input rate under the provider’s pricing schedule, compared with 25% for Hy3 and the same 25% assumed for the Qwen3.5 models and LiteExec.

AuthorsOpenReview ↗

Response to Reviewer uyRT (Part 2)

  • Results also come from single runs, with some search gains being small, and it isn’t clear how well this carries over to other planners or agent setups beyond those tested
  • Do you have repeated runs, or any sense of the noise on the smaller improvements?

Thank you for this feedback. We agree that repeated evaluations help assess whether the reported gains, particularly the smaller improvements, are consistent across runs, and we appreciate the opportunity to quantify this variability.

Evaluation across repeated runs. We supplemented the original results with three repeated evaluation runs for each configuration and benchmark, using Hy3 as the planner in the planner–executor settings. The tables below report mean ± sample standard deviation across three evaluation runs per configuration and benchmark. For each score difference, we compute a nominal 95% Welch tt confidence interval using the three benchmark scores for each configuration.

Table 1. Coding performance

ConfigurationSWE-bench Verified (% resolved)SWE-bench Multilingual (% resolved)
Hy3 (monolithic)73.1 ± 0.771.4 ± 0.2
Hy3 + Qwen3.5-9B66.4 ± 1.259.8 ± 1.9
Hy3 + LiteExec-9B70.6 ± 1.067.0 ± 0.9
Δ (LiteExec-9B − Qwen3.5-9B) [95% CI]+4.2 [1.6, 6.8]+7.2 [3.3, 11.1]

Across the three reported evaluations, LiteExec-9B improved the mean task resolution rate over the Qwen3.5-9B baseline by 4.2 percentage points on SWE-bench Verified and 7.2 percentage points on SWE-bench Multilingual. Both nominal 95% confidence intervals lie entirely above zero, supporting positive gains on both benchmarks.

Table 2. Search performance

ConfigurationBrowseComp (accuracy, %)DeepSearchQA (F1, 0–100)
Hy3 (monolithic)64.3 ± 0.889.5 ± 1.2
Hy3 + Qwen3.5-9B52.7 ± 0.663.7 ± 0.3
Hy3 + LiteExec-9B56.2 ± 0.678.6 ± 2.0
Δ (LiteExec-9B − Qwen3.5-9B) [95% CI]+3.5 [2.1, 4.9]+14.9 [10.1, 19.7]

Across the three reported evaluations, LiteExec-9B improved over the Qwen3.5-9B baseline by 3.5 percentage points in BrowseComp accuracy and 14.9 F1 points on DeepSearchQA. Both nominal 95% confidence intervals lie entirely above zero, including the interval for the smaller BrowseComp gain.

Table 3. Standalone and collaborative performance under different training methods

Model / TrainingSWE-bench VerifiedSWE-bench Multilingual
MonolithicPlan–ExecutionΔ [95% CI]MonolithicPlan–ExecutionΔ [95% CI]
Qwen3.5-9B59.4 ± 0.266.4 ± 1.2+7.0 [4.1, 9.9]50.4 ± 0.559.8 ± 1.9+9.3 [5.1, 13.6]
+ GRPO59.9 ± 1.164.9 ± 1.6+5.0 [1.7, 8.3]51.5 ± 1.458.8 ± 3.1+7.2 [0.8, 13.7]
+ Executor GRPO (LiteExec)55.5 ± 0.170.6 ± 1.0+15.1 [12.6, 17.7]45.3 ± 1.267.0 ± 0.9+21.7 [19.2, 24.1]

LiteExec-9B has the lowest mean standalone scores and the highest mean scores under plan–execution. The gains from plan–execution over standalone evaluation are 15.1 percentage points on SWE-bench Verified and 21.7 percentage points on SWE-bench Multilingual, with both nominal 95% confidence intervals above zero. These results suggest that Executor GRPO improves the ability of the executor to collaborate effectively with the planner.

Taken together, the repeated evaluations and confidence intervals provide evidence that the observed gains in the evaluated configurations are consistent across runs.

We address generalization across planners and agent setups in Part 3.

AuthorsOpenReview ↗

Response to Reviewer uyRT (Part 3)

Continuing from Part 2, we discuss generalization across planners and agent setups.

Evaluation across planners. LiteExec-9B is trained only on coding tasks with Hy3 as the planner. Tables 1 and 2 evaluate its transfer to two additional planners, Qwen3.8-27B and DeepSeek V4 Flash, and to search tasks, without further training. On both coding benchmarks, LiteExec-9B improves over the off-the-shelf Qwen3.5-9B executor under all three planners. On search tasks, it improves performance in five of the six evaluated settings, with an unweighted average gain of 9.6 points across the six reported scores (the only decline is on BrowseComp with Qwen3.8-27B, where accuracy decreases by 3.0 percentage points, from 41.5% to 38.5%). These results support transfer of executor training across both planners and task domains.

Evaluation with an additional agent harness. To further assess whether the benefits of executor training extend beyond the agent setup used in the paper, where only one executor can be active at a time, we implemented a harness variant that allows the planner to invoke multiple executors within a single round (with the planner prompt explicitly permitting parallel delegation). We compare Qwen3.5-9B and LiteExec-9B under concurrency limits of one (as in the paper), two, and four executors. The planner determines whether and when to delegate subtasks in parallel. We use Hy3 as the planner and the same benchmark tasks across settings, and report the mean and sample standard deviation across three evaluation runs for each setting.

Max. concurrent executorsExecutorSWE-bench Verified (% resolved) ↑SWE-bench Multilingual (% resolved) ↑
1Qwen3.5-9B66.4 ± 1.259.8 ± 1.9
1LiteExec-9B70.6 ± 1.067.0 ± 0.9
2Qwen3.5-9B66.9 ± 1.558.6 ± 1.0
2LiteExec-9B69.4 ± 0.963.0 ± 0.3
4Qwen3.5-9B67.6 ± 2.058.2 ± 0.7
4LiteExec-9B69.5 ± 0.664.2 ± 0.7

Across all evaluated agent setups, LiteExec-9B achieves higher mean task resolution rates than the off-the-shelf Qwen3.5-9B executor on both benchmarks. These results suggest that the benefits of executor training extend beyond the agent setup used in the paper to a harness that supports parallel execution.

Reviewer_Hq9wOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors focus on long horizon agents that use one large/frontier model for every step of the process. This is quite expensive. The obvious fix is to use smaller models for specific tasks, while the planner remains a very capable model. However if done naively (generic small model for task execution), this can increase cost due to failures and re-executions. The authors build LiteExec, that uses a 9B model for task execution trained with GRPO. The planner is frozen and uses a large model),and the reward is the end-to-end task outcome. LiteExec reaches 70.0 vs. 73.0 on the baseline on SWE-bench with ~88% fewer large-model tokens.

Critical strengths
  • The solution is simple and works well (freeze planner, reward whole episode, update executor). Seems easy to reproduce as well.

  • Table 3 shows that it outperforms standard GRPO of the model. Executor GRPO does provide a new capabilitiy.

  • The executor transfers from coding to two other types of tasks.

Critical weaknesses
  • The cost efficiency depends on API prices and the effectiveness of caching.

  • This paper can benefit from a SFT or prompt-only baseline.

  • I would like to see experiments that measure variance. It's not clear what is actual improvement vs noise to be honest.

Questions for authors

There are two key questions I would like to know about

  • How well would a prompt only or SFT baseline work for this challenge? E.g., provide stronger instructions to return code diffs test output or a clear blocked status. Or tune the 9B model on execution reports from the bigger model on the same planner. Would either be competitve?

  • Without confidence intervals it is difficult to tell if the results in tables 1, 2, 3, are persistent or just noise. Can you do more training runs or evaluation passes to quantify variance?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Hq9w (Part 1)

We sincerely thank you for your thoughtful suggestions and valuable feedback. We have carefully considered each of your comments and provide a point-by-point response below. We hope these responses help clarify our work and address your concerns.

  • The cost efficiency depends on API prices and the effectiveness of caching.

Thank you for highlighting this important consideration. We agree that cost savings depend on API pricing and caching effectiveness, and appreciate the opportunity to clarify these dependencies from two perspectives.

First, API prices reflect several factors beyond model size, including serving infrastructure and provider pricing policies. For example, although Qwen3.5-35B-A3B has fewer parameters than Hy3-295B-A21B, its listed price per million output tokens was higher (USD 0.80 vs. USD 0.528). Pricing differences also extend to cached input. The cached-input rate was about 3% of the standard input rate for DeepSeek V4 Flash, compared with 25% for Hy3. The provider did not list a separate cached-input rate for Qwen3.5, so we used the same 25% ratio as for Hy3 to estimate caching-adjusted costs for Qwen3.5 models and LiteExec. For each planner, we apply the same token rates to LiteExec and the off-the-shelf Qwen3.5-9B executor. This makes the cost comparison more direct because the estimated difference reflects how the two configurations use tokens at the same rates. Under idealized caching, where all eligible input tokens receive the cached-input rate, LiteExec has lower estimated costs than the off-the-shelf executor across all three planners on both coding benchmarks. The cost comparison with the corresponding monolithic agents varies by planner and depends on relative API prices and caching discounts (e.g., the larger discount for DeepSeek V4 Flash). Appendix C (Table 5) provides the full cost comparison as a reference under the specified API rates and caching discounts. In the next version of the manuscript, we will clarify how API prices and caching discounts affect the cost comparison with monolithic agents.

Second, our pricing sensitivity analysis quantifies when cost savings hold. In Appendix C (“Sensitivity to Pricing”), we use the measured token usage in the plan–execution and monolithic configurations to derive the break-even ratio of the effective per-token price for the large model to that for the small executor model, at which the two configurations have the same estimated cost. The break-even ratios range from approximately 0.91 to 1.19 across the evaluated coding settings. These thresholds are close to one, indicating that cost savings do not require a large per-token price gap between the large model and the small executor model. This analysis complements our API-based estimates by identifying the pricing conditions under which LiteExec is more cost-efficient than monolithic large-model agents.

AuthorsOpenReview ↗

Response to Reviewer Hq9w (Part 2)

  • This paper can benefit from a SFT or prompt-only baseline.

  • How well would a prompt only or SFT baseline work for this challenge? E.g., provide stronger instructions to return code diffs test output or a clear blocked status. Or tune the 9B model on execution reports from the bigger model on the same planner. Would either be competitive?

Thank you for suggesting these baselines. We agree that stronger prompting and supervised fine-tuning are important alternatives to examine. To assess how they compare with LiteExec, we evaluate two additional prompt variants and an SFT baseline, as detailed below.

Stronger instructions. We append the following instructions to the original Qwen3.5-9B executor prompt.

Support your report with concrete evidence. For code changes, include the relevant diff. For tests, include the command you ran and the relevant output, or state that tests were not run. If you are blocked, explain what you tried, what prevented progress, and what you need from the planner to continue. Report only changes and results you actually observed.

Stronger instructions + examples. We retain the instructions above and append two examples.

The following examples illustrate the level of detail to provide. Use evidence from your own work rather than copying the example details.

Example 1: A completed fix

`Status`: `completed`

`Result`: Updated `src/parser.py` so that empty input returns an empty list. The relevant diff is:

```diff
--- a/src/parser.py
+++ b/src/parser.py
@@ -12,2 +12,2 @@
-    if text is None:
+    if not text:
         return []
```

I ran `pytest -q tests/test_parser.py`:

```text
3 passed in 0.12s
```

These tests passed. I did not run the full test suite.

Example 2: A blocked test run

`Status`: `blocked`

`Result`: I tried to run `pytest -q tests/test_parser.py`, but test collection failed:

```text
E   ModuleNotFoundError: No module named 'yaml'
Interrupted: 1 error during collection
```

No tests ran. I also tried to install the missing dependency, but the configured package source was unreachable. To continue, I need access to a working package source or an environment with the dependency installed.

Supervised fine-tuning. We additionally fine-tune Qwen3.5-9B for two epochs on 10,000 trajectories sampled from successful runs on the same training dataset as LiteExec, with Hy3 serving as the executor during data collection.

Results and analysis. The table reports task resolution rates (%).

ExecutorSWE-bench Verified (% resolved)SWE-bench Multilingual (% resolved)
Qwen3.5-9B (original prompt)66.4 ± 1.259.8 ± 1.9
Qwen3.5-9B (stronger instructions)64.7 ± 0.958.4 ± 1.0
Qwen3.5-9B (stronger instructions + examples)66.6 ± 0.457.7 ± 1.2
Qwen3.5-9B (SFT on larger-model trajectories)69.7 ± 1.262.2 ± 0.5
LiteExec-9B70.6 ± 1.067.0 ± 0.9

As shown in the table above, adding stronger instructions, with or without examples, changed the mean task resolution rate of Qwen3.5-9B as an executor by at most 2.1 percentage points relative to the original prompt. These results suggest that prompt refinement alone may be insufficient to elicit effective collaboration from the small executor model. Relative to the original Qwen3.5-9B model, SFT on larger-model trajectories improved the mean resolution rate by 3.3 and 2.4 percentage points on SWE-bench Verified and SWE-bench Multilingual, respectively. Compared with these prompt-only and SFT alternatives, LiteExec-9B achieved higher mean task resolution rates on both benchmarks.

We hypothesize that two factors contribute to the observed performance advantage of LiteExec. First, mainstream post-training objectives emphasize independent task solving. For example, when RLVR rewards a standalone model solely for final task success, it reinforces successful attempts to solve the task independently. The model may learn to keep trying on its own even when it is unsure how to proceed, because another attempt still offers a chance of earning a reward. If the model stops to report a blocker while the task remains unresolved, the report earns no reward for task success, even if it would facilitate collaboration. As a result, collaborative behavior may be difficult to elicit through prompting alone. Second, although SFT on larger-model trajectories improves performance, the capability gap between the teacher and the executor may limit the supervision available for collaboration. A larger model may independently complete steps that present difficulties for a smaller executor, leaving its trajectories without examples of the help-seeking behaviors needed by the smaller model. LiteExec addresses both limitations through dedicated collaboration training with a large-model planner and on-policy RL, enabling the executor to learn collaborative behaviors suited to its own capabilities.

AuthorsOpenReview ↗

Response to Reviewer Hq9w (Part 3)

  • I would like to see experiments that measure variance. It's not clear what is actual improvement vs noise to be honest.

  • Without confidence intervals it is difficult to tell if the results in tables 1, 2, 3, are persistent or just noise. Can you do more training runs or evaluation passes to quantify variance?

Thank you for this feedback. We agree that repeated evaluations and confidence intervals help assess whether the reported gains are consistent across runs, and we appreciate the opportunity to quantify the variability of these results.

We supplemented the original results with three repeated evaluation runs for each configuration and benchmark, using Hy3 as the planner in the planner–executor settings. The tables below report mean ± sample standard deviation across three evaluation runs per configuration and benchmark. For each score difference, we compute a nominal 95% Welch tt confidence interval using the three benchmark scores for each configuration.

Table 1. Coding performance

ConfigurationSWE-bench Verified (% resolved)SWE-bench Multilingual (% resolved)
Hy3 (monolithic)73.1 ± 0.771.4 ± 0.2
Hy3 + Qwen3.5-9B66.4 ± 1.259.8 ± 1.9
Hy3 + LiteExec-9B70.6 ± 1.067.0 ± 0.9
Δ (LiteExec-9B − Qwen3.5-9B) [95% CI]+4.2 [1.6, 6.8]+7.2 [3.3, 11.1]

Across the three reported evaluations, LiteExec-9B improved the mean task resolution rate over the Qwen3.5-9B baseline by 4.2 percentage points on SWE-bench Verified and 7.2 percentage points on SWE-bench Multilingual. Both nominal 95% confidence intervals lie entirely above zero, supporting positive gains on both benchmarks.

Table 2. Search performance

ConfigurationBrowseComp (accuracy, %)DeepSearchQA (F1, 0–100)
Hy3 (monolithic)64.3 ± 0.889.5 ± 1.2
Hy3 + Qwen3.5-9B52.7 ± 0.663.7 ± 0.3
Hy3 + LiteExec-9B56.2 ± 0.678.6 ± 2.0
Δ (LiteExec-9B − Qwen3.5-9B) [95% CI]+3.5 [2.1, 4.9]+14.9 [10.1, 19.7]

Across the three reported evaluations, LiteExec-9B improved over the Qwen3.5-9B baseline by 3.5 percentage points in BrowseComp accuracy and 14.9 F1 points on DeepSearchQA. Both nominal 95% confidence intervals lie entirely above zero, including the interval for the smaller BrowseComp gain.

Table 3. Standalone and collaborative performance under different training methods

Model / TrainingSWE-bench VerifiedSWE-bench Multilingual
MonolithicPlan–ExecutionΔ [95% CI]MonolithicPlan–ExecutionΔ [95% CI]
Qwen3.5-9B59.4 ± 0.266.4 ± 1.2+7.0 [4.1, 9.9]50.4 ± 0.559.8 ± 1.9+9.3 [5.1, 13.6]
+ GRPO59.9 ± 1.164.9 ± 1.6+5.0 [1.7, 8.3]51.5 ± 1.458.8 ± 3.1+7.2 [0.8, 13.7]
+ Executor GRPO (LiteExec)55.5 ± 0.170.6 ± 1.0+15.1 [12.6, 17.7]45.3 ± 1.267.0 ± 0.9+21.7 [19.2, 24.1]

LiteExec-9B has the lowest mean standalone scores and the highest mean scores under plan–execution. The gains from plan–execution over standalone evaluation are 15.1 percentage points on SWE-bench Verified and 21.7 percentage points on SWE-bench Multilingual, with both nominal 95% confidence intervals above zero. These results suggest that Executor GRPO improves the ability of the executor to collaborate effectively with the planner.

Taken together, the repeated evaluations and confidence intervals provide evidence that the observed gains in the evaluated configurations are consistent across runs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.