- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes RECAST, a framework that treats context construction as sequential evidence routing over heterogeneous sources. A lightweight RouterLM iteratively invokes lexical, semantic, or relational primitives, or requests customized programs from a frozen CompilerLM, and decides when evidence suffices for a frozen AnswerLM. The router is trained with SFT followed by GRPO.
Critical strengths
- The paper is generally clearly written and easy to follow. The overall pipeline, the action space of the RouterLM, and the roles of the three models are described in sufficient detail.
- The problem is clear. The paper points out that in many tasks the required evidence must be derived through filtering, aggregation, or computation rather than directly retrieved, and illustrates this with a concrete example.
Critical weaknesses
-
Limited novelty. General-purpose coding agents such as Codex and Claude Code can already query data iteratively, write and execute programs, inspect outputs, and decide when to stop. RECAST appears to be a constrained instance of this paradigm, and the paper should clarify what is fundamentally new.
-
Missing key baselines. All baselines are training-free, and Direct Code generates a single program without execution feedback. The authors should at least compare with Search-R1 [1], a trained retrieval agent, and CodeAct [2], a multi-turn code-action agent, using the same backbone, tools, and budget.
-
Generalization across router backbones is untested. SFT+GRPO is applied only to Qwen3.5-9B, whose own trajectories also form the SFT data. Training additional routers of different scales and model families would show whether the training gains are model-specific.
[1] Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning.
[2] Executable Code Actions Elicit Better LLM Agents.
Questions for authors
see weakness
5 comments
Response to C4NU 2
---continues from previous comment---
- Learnability: whether a small model can do it.
- Coding agent: Its capability comes from the frontier model. A small model could in principle imitate its trajectories, but there is no evidence that this succeeds, and the trajectories are long, free-form transcripts (48.7k to 71.1k tokens per question) that make imitation difficult.
- RECAST: The structured action space gives a small model a compact, well-defined policy to learn directly from task outcomes, without a frontier teacher for routing. SFT and GRPO raise the 9B router from 64.1% to 75.6%, which is within 1.7% of Codex/GPT-6 Luna with 3.7× fewer tokens, and surpasses a training-free Gemini 3.5 Flash router (70.6%). Structure matters for learning: the multi-turn tool agent Search-R1 trained with the same GRPO data and reward reaches only 67.2%, compared with 72.6% for GRPO-only RECAST. The learned policy also generalizes zero-shot to held-out benchmarks and transfers across compilers.
To summarize, coding agents reach high accuracy by running an unconstrained loop on frontier models. RECAST identifies the structure that makes evidence construction work and shows that, with this structure, a trained 9B router approaches frontier-agent accuracy at a fraction of the compute.
Efficiency. RECAST uses 19.4k tokens per question, compared with 48.7k to 71.1k for Codex, and most of its tokens are processed by the 9B router.
Openness and reproducibility. Results from closed coding agents depend on proprietary models that change without notice and cannot be retrained or audited. RECAST’s trained component is an open 9B router, and its other components are swappable: Section 4.4.2 shows that the trained policy transfers across compilers, including the open Qwen3.5-9B.
To summarize, coding agents show that evidence construction is achievable with frontier models, while RECAST provides a principled, learnable methodology for it, with low cost and open, swappable components as additional advantages.
C4NU-W2: Missing key baselines. All baselines are training-free, and Direct Code generates a single program without execution feedback. The authors should at least compare with Search-R1 [1], a trained retrieval agent, and CodeAct [2], a multi-turn code-action agent, using the same backbone, tools, and budget.
We conduct additional experiments including 1) multi-turn Direct Code, 2) existing multi-turn code-action agent CodeAct, and 3) trained retrieval agent Search-R1, and show that RECAST still outperforms the baselines.
Specifically, we
- extend Direct Code to multi-turn execution, allowing it to inspect execution results and errors and revise its program for up to six rounds (same as RECAST)
- supply CodeAct agent, which writes and executes Python code actions over the source records in a multi-turn loop, with the same base model, tools, compiler and answerer as RECAST.
- supply Search-R1 agent with same base model, tools, compiler and answerer as RECAST. It interleaves free-form reasoning with tool calls.
- train the Search-R1 agent with GRPO from raw Qwen3.5-9B, using the same training data, reward, steps, frozen answerer and judge as GRPO-only RECAST.
| Method | DataBench | FinQA | HiTab | HotpotQA | LaMP | MultiHiertt | Avg. |
|---|---|---|---|---|---|---|---|
| Multi-turn Direct Code | 87.0 | 8.0 | 31.0 | 66.0 | 64.0 | 4.0 | 43.3 |
| Search-R1 | 70.0 | 58.0 | 36.0 | 92.0 | 46.0 | 56.0 | 59.7 |
| Search-R1 (GRPO) | 83.0 | 59.0 | 50.0 | 92.0 | 56.0 | 63.0 | 67.2 |
| CodeAct | 67.0 | 55.0 | 37.0 | 94.0 | 47.0 | 53.0 | 58.8 |
| RECAST(Training Free) | 86.0 | 57.0 | 60.7 | 83.0 | 68.3 | 29.3 | 64.1 |
| RECAST(GRPO) | 81.7 | 64.0 | 75.7 | 88.3 | 64.0 | 61.7 | 72.6 |
From the results, we can observe that with same Qwen 3.5 9B as the backbone, multi-turn Direct Code achieves 43.3%, CodeAct achieves 58.8%, and Search-R1 achieves 59.7%. Training-free RECAST achieves 64.1%, still outperforms all previous training-free baselines. While Search-R1 (GRPO) achieves 67.2%, a significant 7.5% improvements over its own training free results, it still is 5.4% lower than RECAST(GRPO) (72.6%).
These results indicate that the performance gap is not explained solely by multi-turn execution feedback or training. Selecting operations, formulating each operation, judging when the evidence suffices, and how the evidence is built all are important components that make RECAST effective.
Response to C4NU 3
C4NU-W3: Generalization across router backbones is untested. SFT+GRPO is applied only to Qwen3.5-9B, whose own trajectories also form the SFT data. Training additional routers of different scales and model families would show whether the training gains are model-specific.
We conduct additional experiments to train additional routers of different scales and model families:
- For different scale, we trained Qwen3.5-4B, which shares the same model family of our reported Qwen3.5-9B, but only has 4B parameters.
- For different family, we trained Llama-3.1-8B-Instruct, which has close parameter size to our reported Qwen3.5-9B, but from a different model family.
We train both models using the same training data, reward, steps, frozen answerer and judge as RECAST (SFT+GRPO). We also tested same strong iterative and agentic retrieval baselines using these two models as the backbone to better show the performance of RECAST over baselines.
Backbone: Qwen3.5-4B
| Method | DataBench | FinQA | HiTab | HotpotQA | LaMP | MultiHiertt | Avg. |
|---|---|---|---|---|---|---|---|
| IRCoT | 39.0 | 30.0 | 57.0 | 93.0 | 69.0 | 56.0 | 57.3 |
| Interact-RAG | 19.0 | 49.0 | 46.0 | 81.0 | 71.0 | 53.0 | 53.2 |
| RECAST (SFT+GRPO) | 86.0 | 59.0 | 73.0 | 80.0 | 66.0 | 56.0 | 70.0 |
Backbone: Llama-3.1-8B-Instruct
| Method | DataBench | FinQA | HiTab | HotpotQA | LaMP | MultiHiertt | Avg. |
|---|---|---|---|---|---|---|---|
| IRCoT | 38.0 | 34.0 | 55.0 | 91.0 | 74.0 | 58.0 | 58.3 |
| Interact-RAG | 40.0 | 35.0 | 52.0 | 67.0 | 66.0 | 58.0 | 53.0 |
| RECAST (SFT+GRPO) | 83.0 | 63.0 | 76.0 | 80.0 | 75.0 | 62.0 | 73.2 |
From the results we can observe that, after SFT+GRPO, RECAST achieves average success rates of 70.0% and 73.2%, respectively, compared with 57.3% and 58.3% for the strongest baselines using the corresponding backbone.
Compared with the original Qwen3.5-9B RECAST at 75.6%, the two backbones retain strong performance at 70.0% and 73.2%, despite using SFT trajectories and a GRPO question pool constructed from the original Qwen3.5-9B model’s behavior rather than tailored to either backbone.
These results demonstrate that RECAST generalizes across router scales and model families, with training data collected using Qwen3.5-9B remaining effective for both a smaller model and a different backbone family.
[1] Jin, Bowen, et al. "Search-r1: Training llms to reason and leverage search engines with reinforcement learning." arXiv preprint arXiv:2503.09516 (2025). [2] Wang, Xingyao, et al. "Executable code actions elicit better llm agents." arXiv preprint arXiv:2402.01030 (2024).
Response to C4NU 1
We thank the reviewer for the constructive comments and helpful feedback! You brought up some great questions and suggestions, which have helped improve our work. We address the reviewer's concerns below.
C4NU-W1: Limited novelty. General-purpose coding agents such as Codex and Claude Code can already query data iteratively, write and execute programs, inspect outputs, and decide when to stop. RECAST appears to be a constrained instance of this paradigm, and the paper should clarify what is fundamentally new.
We claim that RECAST's novelty and contribution are not negated by the fact that general-purpose coding agents can query data, write and execute code, inspect outputs, and decide when to stop.
Coding agents like Codex or Claude Code are powerful. However, we show RECAST's novelty and contribution over them with:
Direct comparison. We evaluated Codex with three frontier backbones. Each prepared evidence for the same Gemini 3.5 Flash AnswerLM and was scored by the same judge on the same test set.
Frontier coding agents achieve higher average accuracy, which is expected given that every decision is made by a frontier model. RECAST, whose only trained component is a 9B router, comes within 1.7 points of Codex/GPT-6 Luna and matches or exceeds it on 3 out of 6 benchmarks.
General-purpose capability ≠ principled methodology. Codex's results show that a frontier model in an unconstrained agent loop can construct useful evidence. They do not show how evidence construction should be structured, which operations it requires, or whether a smaller model can learn to do it. Because Codex is a closed system, these questions cannot be studied by examining or modifying it. RECAST answers each of them:
--continues in next comment---