acceptodds
Under review as a conference paper at ICLR 2027

RECAST: Learning to Compute the Right Context through Adaptive Evidence Routing

Abstract

Large language models are increasingly applied to tasks grounded in long, heterogeneous information sources. Conventional Retrieval-Augmented Generation (RAG) relies on fixed similarity-based retrieval, while agentic variants adapt queries and tool use but remain largely retrieval-centric. However, in many tasks, the evidence required for a solution is not explicitly present in any single source item. Instead, it must be derived through filtering, aggregation, or computation across multiple source items. In this work, we introduce RECAST (**R**outing **E**vidence through **C**omputation, **A**ccess, and **S**ynthesized **T**ools), a learned framework that formulates evidence construction as a sequential decision process over heterogeneous retrieval and computation operations, allowing evidence to be actively derived rather than merely retrieved. A lightweight RouterLM iteratively selects and formulates primitive operations or specifies customized operations for a frozen CompilerLM to translate into executable code. Once it judges the evidence sufficient, RouterLM passes the accepted evidence to a frozen AnswerLM to produce the final solution. We train RouterLM with supervised fine-tuning (SFT) followed by group relative policy optimization (GRPO). Across six heterogeneous benchmark families, RECAST achieves a mean success rate of 75.6%, outperforming the strongest large-model baseline by 15.9%. Moreover, training enables the Qwen3.5-9B RouterLM to outperform a training-free Gemini 3.5 Flash RouterLM by 5.0%. On three held-out benchmarks, RECAST improves over the strongest baseline by 15.0% on average, demonstrating strong zero-shot generalization across tasks and heterogeneous source representations.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Reasoning, Retrieval-Augmented Generation, LLM Agent

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_C4NUOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RECAST, a framework that treats context construction as sequential evidence routing over heterogeneous sources. A lightweight RouterLM iteratively invokes lexical, semantic, or relational primitives, or requests customized programs from a frozen CompilerLM, and decides when evidence suffices for a frozen AnswerLM. The router is trained with SFT followed by GRPO.

Critical strengths
  1. The paper is generally clearly written and easy to follow. The overall pipeline, the action space of the RouterLM, and the roles of the three models are described in sufficient detail.
  2. The problem is clear. The paper points out that in many tasks the required evidence must be derived through filtering, aggregation, or computation rather than directly retrieved, and illustrates this with a concrete example.
Critical weaknesses
  1. Limited novelty. General-purpose coding agents such as Codex and Claude Code can already query data iteratively, write and execute programs, inspect outputs, and decide when to stop. RECAST appears to be a constrained instance of this paradigm, and the paper should clarify what is fundamentally new.

  2. Missing key baselines. All baselines are training-free, and Direct Code generates a single program without execution feedback. The authors should at least compare with Search-R1 [1], a trained retrieval agent, and CodeAct [2], a multi-turn code-action agent, using the same backbone, tools, and budget.

  3. Generalization across router backbones is untested. SFT+GRPO is applied only to Qwen3.5-9B, whose own trajectories also form the SFT data. Training additional routers of different scales and model families would show whether the training gains are model-specific.

[1] Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning.

[2] Executable Code Actions Elicit Better LLM Agents.

Questions for authors

see weakness

5 comments
AuthorsOpenReview ↗

Response to C4NU 1

We thank the reviewer for the constructive comments and helpful feedback! You brought up some great questions and suggestions, which have helped improve our work. We address the reviewer's concerns below.

C4NU-W1: Limited novelty. General-purpose coding agents such as Codex and Claude Code can already query data iteratively, write and execute programs, inspect outputs, and decide when to stop. RECAST appears to be a constrained instance of this paradigm, and the paper should clarify what is fundamentally new.

We claim that RECAST's novelty and contribution are not negated by the fact that general-purpose coding agents can query data, write and execute code, inspect outputs, and decide when to stop.

Coding agents like Codex or Claude Code are powerful. However, we show RECAST's novelty and contribution over them with:

  • General-purpose capability ≠ principled methodology (central contribution). A coding agent shows that a frontier model, given an open-ended harness, can produce useful evidence for answer generation. However, 1) it does not explain what structure makes evidence construction work, 2) it does not offer a method that can be analyzed, trained, or transferred. RECAST contributes this methodology: a formulation of evidence construction as a sequential decision process over a typed operation space, an analysis of which components matter, and a training procedure that instills the policy in a 9B model.
  • Efficiency. With this methodology, a 9B router reaches near-frontier-agent accuracy with 2.5 to 3.7× fewer tokens.
  • Openness and reproducibility. RECAST is trainable, inspectable, and can run entirely on open models. Closed coding agents are none of these.

Direct comparison. We evaluated Codex with three frontier backbones. Each prepared evidence for the same Gemini 3.5 Flash AnswerLM and was scored by the same judge on the same test set.

MethodDataBenchFinQAHiTabHotpotQALaMPMultiHierttAvg.Tokens/qRel. tokens
Codex, GPT-6 Luna86.074.076.098.064.066.077.371.1k3.7×
Codex, GPT-6.1 Terra87.078.082.097.069.066.079.848.7k2.5×
Codex, GPT-6.1 Sol93.074.080.099.078.067.081.863.7k3.3×
RECAST, Qwen3.5-9B (SFT+GRPO)83.368.376.392.068.365.375.619.4k1.0×

Frontier coding agents achieve higher average accuracy, which is expected given that every decision is made by a frontier model. RECAST, whose only trained component is a 9B router, comes within 1.7 points of Codex/GPT-6 Luna and matches or exceeds it on 3 out of 6 benchmarks.

General-purpose capability ≠ principled methodology. Codex's results show that a frontier model in an unconstrained agent loop can construct useful evidence. They do not show how evidence construction should be structured, which operations it requires, or whether a smaller model can learn to do it. Because Codex is a closed system, these questions cannot be studied by examining or modifying it. RECAST answers each of them:

  • Structure: what evidence construction requires.
    • Coding agent: The agent can run any command and write any code. Which operations to use, what each step should produce, and when to stop are all left implicit in the frontier model's behavior.
    • RECAST: We define evidence construction as a sequential decision process over a typed operation space: parameterized primitives (lexical, semantic, relational) + synthesized code. The router formulates each operation (evidence goal, required output) precisely enough that a compiler without access to the task can execute it, and explicitly decides when the evidence suffices for the AnswerLM.
    • Evidence: Without this structure, an unconstrained loop depends on model scale. A Codex-style code-action agent (CodeAct[2]) with the same 9B model, tools, compiler and answerer achieves 58.8%, compared with 64.1% for training-free RECAST and 75.6% for trained RECAST.(Please see details below in C4NU-W2)

--continues in next comment---

AuthorsOpenReview ↗

Response to C4NU 2

---continues from previous comment---

  • Learnability: whether a small model can do it.
    • Coding agent: Its capability comes from the frontier model. A small model could in principle imitate its trajectories, but there is no evidence that this succeeds, and the trajectories are long, free-form transcripts (48.7k to 71.1k tokens per question) that make imitation difficult.
    • RECAST: The structured action space gives a small model a compact, well-defined policy to learn directly from task outcomes, without a frontier teacher for routing. SFT and GRPO raise the 9B router from 64.1% to 75.6%, which is within 1.7% of Codex/GPT-6 Luna with 3.7× fewer tokens, and surpasses a training-free Gemini 3.5 Flash router (70.6%). Structure matters for learning: the multi-turn tool agent Search-R1 trained with the same GRPO data and reward reaches only 67.2%, compared with 72.6% for GRPO-only RECAST. The learned policy also generalizes zero-shot to held-out benchmarks and transfers across compilers.

To summarize, coding agents reach high accuracy by running an unconstrained loop on frontier models. RECAST identifies the structure that makes evidence construction work and shows that, with this structure, a trained 9B router approaches frontier-agent accuracy at a fraction of the compute.

Efficiency. RECAST uses 19.4k tokens per question, compared with 48.7k to 71.1k for Codex, and most of its tokens are processed by the 9B router.

Openness and reproducibility. Results from closed coding agents depend on proprietary models that change without notice and cannot be retrained or audited. RECAST’s trained component is an open 9B router, and its other components are swappable: Section 4.4.2 shows that the trained policy transfers across compilers, including the open Qwen3.5-9B.

To summarize, coding agents show that evidence construction is achievable with frontier models, while RECAST provides a principled, learnable methodology for it, with low cost and open, swappable components as additional advantages.

C4NU-W2: Missing key baselines. All baselines are training-free, and Direct Code generates a single program without execution feedback. The authors should at least compare with Search-R1 [1], a trained retrieval agent, and CodeAct [2], a multi-turn code-action agent, using the same backbone, tools, and budget.

We conduct additional experiments including 1) multi-turn Direct Code, 2) existing multi-turn code-action agent CodeAct, and 3) trained retrieval agent Search-R1, and show that RECAST still outperforms the baselines.

Specifically, we

  • extend Direct Code to multi-turn execution, allowing it to inspect execution results and errors and revise its program for up to six rounds (same as RECAST)
  • supply CodeAct agent, which writes and executes Python code actions over the source records in a multi-turn loop, with the same base model, tools, compiler and answerer as RECAST.
  • supply Search-R1 agent with same base model, tools, compiler and answerer as RECAST. It interleaves free-form reasoning with tool calls.
  • train the Search-R1 agent with GRPO from raw Qwen3.5-9B, using the same training data, reward, steps, frozen answerer and judge as GRPO-only RECAST.
MethodDataBenchFinQAHiTabHotpotQALaMPMultiHierttAvg.
Multi-turn Direct Code87.08.031.066.064.04.043.3
Search-R170.058.036.092.046.056.059.7
Search-R1 (GRPO)83.059.050.092.056.063.067.2
CodeAct67.055.037.094.047.053.058.8
RECAST(Training Free)86.057.060.783.068.329.364.1
RECAST(GRPO)81.764.075.788.364.061.772.6

From the results, we can observe that with same Qwen 3.5 9B as the backbone, multi-turn Direct Code achieves 43.3%, CodeAct achieves 58.8%, and Search-R1 achieves 59.7%. Training-free RECAST achieves 64.1%, still outperforms all previous training-free baselines. While Search-R1 (GRPO) achieves 67.2%, a significant 7.5% improvements over its own training free results, it still is 5.4% lower than RECAST(GRPO) (72.6%).

These results indicate that the performance gap is not explained solely by multi-turn execution feedback or training. Selecting operations, formulating each operation, judging when the evidence suffices, and how the evidence is built all are important components that make RECAST effective.

AuthorsOpenReview ↗

Response to C4NU 3

C4NU-W3: Generalization across router backbones is untested. SFT+GRPO is applied only to Qwen3.5-9B, whose own trajectories also form the SFT data. Training additional routers of different scales and model families would show whether the training gains are model-specific.

We conduct additional experiments to train additional routers of different scales and model families:

  • For different scale, we trained Qwen3.5-4B, which shares the same model family of our reported Qwen3.5-9B, but only has 4B parameters.
  • For different family, we trained Llama-3.1-8B-Instruct, which has close parameter size to our reported Qwen3.5-9B, but from a different model family.

We train both models using the same training data, reward, steps, frozen answerer and judge as RECAST (SFT+GRPO). We also tested same strong iterative and agentic retrieval baselines using these two models as the backbone to better show the performance of RECAST over baselines.

Backbone: Qwen3.5-4B

MethodDataBenchFinQAHiTabHotpotQALaMPMultiHierttAvg.
IRCoT39.030.057.093.069.056.057.3
Interact-RAG19.049.046.081.071.053.053.2
RECAST (SFT+GRPO)86.059.073.080.066.056.070.0

Backbone: Llama-3.1-8B-Instruct

MethodDataBenchFinQAHiTabHotpotQALaMPMultiHierttAvg.
IRCoT38.034.055.091.074.058.058.3
Interact-RAG40.035.052.067.066.058.053.0
RECAST (SFT+GRPO)83.063.076.080.075.062.073.2

From the results we can observe that, after SFT+GRPO, RECAST achieves average success rates of 70.0% and 73.2%, respectively, compared with 57.3% and 58.3% for the strongest baselines using the corresponding backbone.

Compared with the original Qwen3.5-9B RECAST at 75.6%, the two backbones retain strong performance at 70.0% and 73.2%, despite using SFT trajectories and a GRPO question pool constructed from the original Qwen3.5-9B model’s behavior rather than tailored to either backbone.

These results demonstrate that RECAST generalizes across router scales and model families, with training data collected using Qwen3.5-9B remaining effective for both a smaller model and a different backbone family.

[1] Jin, Bowen, et al. "Search-r1: Training llms to reason and leverage search engines with reinforcement learning." arXiv preprint arXiv:2503.09516 (2025). [2] Wang, Xingyao, et al. "Executable code actions elicit better llm agents." arXiv preprint arXiv:2402.01030 (2024).

Reviewer_ckhCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RECAST, a system that builds evidence through both retrieval and computation. A RouterLM repeatedly chooses the next action. It can use lexical retrieval, semantic retrieval, a relational SQL operation, custom code synthesis, or stop and accept the current evidence. A frozen CompilerLM converts a requested custom operation into code, and a frozen AnswerLM gives the final answer.

The RouterLM is trained with supervised fine-tuning and GRPO. The paper evaluates the system on six benchmark families and three held-out datasets. It also studies different judges, different compiler models, tool usage, and token cost. The final system reaches an average success rate of 75.6% on the six main benchmarks.

Critical strengths
  1. The paper starts from a reasonable and useful problem. Traditional RAG mainly searches for existing text. Many table, finance, and data analysis questions instead require filtering, joins, aggregation, or a new calculation. The retrieve-or-compute view is simple and meaningful.

  2. The empirical results are strong and the paper provides useful analysis. The authors test several data formats and task types. They also use alternative judges, token F1, different compiler models, action statistics, and token-cost analysis. The main performance gain remains with other judges, which is a useful check.

Critical weaknesses
  1. The main baselines do not have the same tools as RECAST. RECAST can use SQL, generated Python code, execution feedback, and up to six rounds. The main retrieval baselines do not receive the same action space. On table and dataframe tasks, tool access itself may explain a large part of the improvement. Therefore, the current comparison does not fully prove that learned routing is better.

  2. A simple routing baseline is missing. Tool usage is strongly related to the dataset type. For example, code synthesis is common on DataBench and almost unused on HotpotQA. A simple fixed rule based on source format may already perform well. Also, the held-out datasets have close training neighbors, so I would describe this as transfer to new datasets rather than a completely new task distribution. The paper also does not report training-seed variation or confidence intervals over test samples.

Questions for authors
  1. Can the authors give the main baselines the same SQL tool, code-synthesis tool, number of rounds, and evidence budget? How much improvement remains after matching the tools?

  2. Can the authors compare with a simple fixed router based on the source type? For example, it can use code for dataframes, relational queries for tables, and semantic retrieval for text passages.

  3. Can the authors report uncertainty over test questions and results from more than one SFT or GRPO training seed? Are the smaller ablation differences stable?

  4. Can the authors make the generalization claim more specific? The held-out datasets are new, but their task types and source formats are close to datasets used during training.

2 comments
AuthorsOpenReview ↗

Response to ckhC 1

We thank the reviewer for the constructive comments and helpful feedback! You brought up some great questions and suggestions, which have helped improve our work. We address the reviewer's concerns below.

ckhC-W1+Q1: The main baselines do not have the same tools as RECAST. RECAST can use SQL, generated Python code, execution feedback, and up to six rounds. The main retrieval baselines do not receive the same action space. On table and dataframe tasks, tool access itself may explain a large part of the improvement. Therefore, the current comparison does not fully prove that learned routing is better. Can the authors give the main baselines the same SQL tool, code-synthesis tool, number of rounds, and evidence budget? How much improvement remains after matching the tools?

To address the concern, we augment IRCoT and Interact-RAG, the strongest iterative and agentic retrieval baselines, with same capabilities as RECAST including retrieval, SQL (new), python code generation with same frozen compiler(new), execution feedback, and up to six rounds.

MethodDataBenchFinQAHiTabHotpotQALaMPMultiHiertTAvg.
Tool-augmented IRCoT85.029.028.072.070.058.057.0
Tool-augmented Interact-RAG76.041.048.059.065.050.056.5
RECAST (Training-Free)86.057.060.783.068.329.364.1

From the results, the augmented IRCoT and Interact-RAG achieve 57.0% and 56.5%, respectively. Training-free RECAST achieves 64.1%, outperforms Tool-augmented IRCoT and Interact-RAG. These results indicate that the performance gap is not explained solely by the original baselines’ lack of computational tools. We explain the core differences in detail below:

Comparing RECAST with the existing iterative and agentic retrieval baselines, we claim two core difference / contribution:

  • Tool set expansion: we broaden evidence construction beyond retrieval, identify retrieval and computation as complementary means, and develop RECAST that supports adaptive tool routing that supports evidence constrution over heterogeneous sources
  • Evidence-construction policy design: The RouterLM does more than selecting operations. In addition, it also formulates the operation and decides when the evidence suffices. It does this over an evidence state that the framework maintains and checks against source records, rather than over a free-form transcript.

Since we augment the baselines with the tool set expansion, the remaining improvements are explained by the Evidence-construction policy design. Especially, since we also augment the baselines same evidence state construction as RECAST, the core difference is mostly "The RouterLM does more than selecting operations. In addition, it also formulates the operation and decides when the evidence suffices."

For our tool augmentation of baselines, the augmented agent picks a tool and writes only its input (a query, SQL, or a free-text code request), without RECAST's structured formulation (evidence goal, required output, reason) or its guidance on choosing operations and stopping. Improvements on DataBench demonstrates the value of code synthesis augmentation, but tool set expansion without structured formulation and guidance resulted in distractions in retrieval, malformed structured tool calls, or inappropriate stopping. This shows that selecting a tool and writing its input is not enough; formulating each operation and judging when the evidence suffices is what makes the tools effective.

ckhC-W2(part1)+Q2: A simple routing baseline is missing. Tool usage is strongly related to the dataset type. For example, code synthesis is common on DataBench and almost unused on HotpotQA. A simple fixed rule based on source format may already perform well. Can the authors compare with a simple fixed router based on the source type? For example, it can use code for dataframes, relational queries for tables, and semantic retrieval for text passages.

We evaluated a source-type-based fixed router implemented as a multi-turn tool-use agent with the same tool implementations, action format, CompilerLM, and AnswerLM as RECAST. It restricts each source type to a predetermined operation category: code synthesis for DataBench, SQL for FinQA and HiTab, and retrieval for HotpotQA, LaMP, and MultiHiertT. Within that category, the agent can still formulate task-specific arguments and revise its requests over multiple rounds.

MethodDataBenchFinQAHiTabHotpotQALaMPMultiHiertTAvg.
Fixed-Tool RECAST (Training-Free)88.044.054.015.066.060.054.5
RECAST (Training-Free)86.057.060.783.068.329.364.1

---continues in the next comment---

AuthorsOpenReview ↗

Response to ckhC 2

---continues from the previous comment---

This baseline achieves 54.5% average success, compared with 64.1% for training-free RECAST. Thus, even without RouterLM training, allowing operation selection to adapt to the question and intermediate outcomes outperforms this source-type-based routing rule. While aggregate tool usage correlates with benchmark type, these results indicate that this correlation alone does not provide an equally effective routing policy.

ckhC-W2(part2)+Q3: The paper also does not report training-seed variation or confidence intervals over test samples Can the authors report uncertainty over test questions and results from more than one SFT or GRPO training seed? Are the smaller ablation differences stable?

To address this concern, we

  • conducted two additional SFT+GRPO training runs with different seeds, keeping the training data, hyperparameters, and checkpoint-selection procedure unchanged
  • computed 95% confidence intervals by resampling test questions within each benchmark
MethodDataBenchFinQAHiTabHotpotQALaMPMultiHiertTAvg.
RECAST (SFT+GRPO, seed 40)87.066.069.084.071.064.073.5
RECAST (SFT+GRPO, seed 41)88.070.075.086.070.063.075.3
RECAST (SFT+GRPO, seed 42)83.069.076.095.067.063.075.5

Across three training seeds, RECAST achieves an average success rate of 74.8 ± 1.1% (mean ± sample standard deviation). Both SFT and GRPO are independently rerun using the same checkpoint-selection procedure. We want to emphasize that we reuse the original GRPO dataset constructed using the seed 42 SFT checkpoint rather than regenerate it for each seed. The strong results suggest that this curriculum remains useful for independently trained checkpoints and does not need to be tailored to each checkpoint’s rollout behavior. Although the average is slightly below the originally reported 75.6%, it remains substantially above the strongest baseline at 59.7%, indicating that the main finding is not specific to one training seed.

Uncertainty over test questions. We use bootstrap resampling to examine whether the observed advantages depend on the particular mix of test questions. We repeatedly sample questions with replacement within each benchmark and compare both methods on exactly the same sampled questions. We perform 100,000 resamples and report 95% percentile bootstrap confidence intervals.

On the full 600-question test set, the original seed-42 RECAST checkpoint achieves 75.5%, with a 95% confidence interval of [72.2%, 78.8%]. Its improvements over the training-free RECAST and strong baselines are:

Comparison on 600 questionsRECAST successComparator successRECAST advantage95% CI for advantage
vs. training-free RECAST75.5%64.5%+11.0 points[7.8, 14.3]
vs. IRCoT75.5%56.3%+19.2 points[13.8, 24.2]
vs. Interact-RAG75.5%54.7%+20.8 points[15.8, 25.8]

All three intervals remain above zero. Thus, the benefits over the training-free RECAST and over both retrieval baselines remain clear when the test questions are resampled. These results support that the observed advantages are not simply due to a favorable mix of questions.

ckhC-W2(part3)+Q4: Also, the held-out datasets have close training neighbors, so I would describe this as transfer to new datasets rather than a completely new task distribution. Can the authors make the generalization claim more specific? The held-out datasets are new, but their task types and source formats are close to datasets used during training.

We thank the reviewer for suggesting a more precise characterization of generalization. We agree that the held-out benchmarks share broad task types and source formats with the training benchmarks. Our held-out experiment is intended to evaluate zero-shot cross-dataset transfer. We have made this scope more explicit in the abstract, introduction, and Section 4.4.1 of the revised paper (all changes labeled blue).

Within these related task families, the held-out datasets nevertheless differ in question construction and evidence organization. For example, 2WikiMultiHopQA uses relation-based question construction and composed reasoning chains, WikiTableQuestions differs from HiTab in table structure and question construction, and TAT-QA presents a different mixture of extraction and numerical reasoning questions over table–text contexts. Without additional training, RECAST achieves 79.3% average success across these benchmarks, compared with 64.3% for the strongest evaluated baseline. These results support the transferability of the learned evidence-construction policy beyond the training datasets.

Reviewer_N2XkOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RECAST addresses context construction over heterogeneous sources by treating retrieval and computation as complementary evidence-producing operations. A Router LM iteratively selects and formulates lexical, semantic, or relational operations, requests customized Python programs from a frozen Compiler LM, and decides when the accumulated evidence is sufficient for a frozen Answer LM. Only the router is trained, using supervised fine-tuning followed by GRPO.

The paper evaluates the framework on six benchmark families and three held-out benchmarks. The trained system achieves an average in-domain success rate of 75.6%, compared with 59.7% for the strongest reported baseline and 64.1% for the same framework with an untrained Qwen router. Additional experiments examine operation-space restrictions, training stages, reward design, alternative judges, compiler transfer, and inference token usage.

Critical strengths

S1. A meaningful problem and a coherent framework. The paper addresses a meaningful limitation of retrieval-centric context construction: relevant evidence may require aggregation, arithmetic, or other transformations rather than simply locating existing passages. The framework provides a clear interface for combining retrieval, relational operations, and synthesized programs across heterogeneous sources. Its evaluation across multiple source formats makes the proposed approach practically interesting.

S2. Useful evidence for the effectiveness of router training. The within-framework comparisons keep the surrounding framework fixed and show that SFT+GRPO raises average success from 64.1% to 75.6%, outperforming either training stage alone. The alternative-judge evaluations and compiler-transfer experiments further support the robustness of the reported system-level gains.

Critical weaknesses

W1: The boundary of the method's novelty relative to existing tool agents is not clearly established. Unifying retrieval, relational queries, and code synthesis under one trainable evidence-construction policy has system value. However, multi-turn tool execution, adjusting actions from execution feedback, and producing intermediate results through data transformations all have precedents, for example in CodeAct and Chain-of-Table. As presented, the paper does not clearly explain what substantive mechanistic difference "constructing evidence rather than solving directly" provides beyond integrating and restating existing capabilities.

W2: The main comparisons do not separate the contribution of the method design from the contribution of tool and interaction capabilities. The retrieval baselines lack equivalent code computation, and Direct Code is restricted to single-shot generation without revision from execution feedback, while RECAST combines retrieval, computation, and multi-round feedback (Appendix C.1). The appendix itself notes that part of Direct Code's failures stem from output-interface and program-execution issues (Appendix E.1), and the operation-space ablations in Table 2 use untrained routers. The results therefore support RECAST being better than the evaluated configurations, but not an advantage of its specific routing and role division over existing multi-turn tool agents. The trained/untrained comparison shows that training helps, but does not resolve this attribution question.

Questions for authors

Q1. If an existing multi-turn tool agent were given the same retrieval, SQL, and Python interfaces, what would be RECAST's most essential mechanistic difference from it? Which of the reported results support the value of that difference?

Q2. What is the operational distinction between "constructing evidence" and "solving directly"? In Figure 1, for example, the synthesized program already computes the requested period; what additional capability is demonstrated by passing its result to the Answer LM as evidence, rather than returning it directly?

Q3. How much of Direct Code's failure rate comes from interface compliance or execution errors, and how much from task-reasoning errors? Does the qualitative analysis in Appendix E.1 imply that its lower scores partly reflect the single-shot setup?

Q4. What is the primary purpose of separating Router and Compiler: reducing cost, improving planning quality, or improving transferability? Which part of the current analysis most directly supports this design choice?

6 comments
AuthorsOpenReview ↗

Response to N2Xk 1

We thank the reviewer for the constructive comments and helpful feedback! You brought up some great questions and suggestions, which have helped improve our work. We address the reviewer's points below (reordered slightly by theme for a clearer flow).

N2Xk-Q2: " What is the operational distinction between "constructing evidence" and "solving directly"? In Figure 1, for example, the synthesized program already computes the requested period; what additional capability is demonstrated by passing its result to the Answer LM as evidence, rather than returning it directly?" W1(part2): As presented, the paper does not clearly explain what substantive mechanistic difference "constructing evidence rather than solving directly" provides beyond integrating and restating existing capabilities.

To address this concern, we would like to first clarify that the computed value in Figure 1 is one of many forms evidence can take, not the defining one. In many tasks the evidence is not itself an answer. Examples include passages that must be composed (HotpotQA), a user's historical examples that inform a personalized judgment (LaMP), and operands combined with a retrieved condition, unit or scale (FinQA, HiTab). Even when a computed value is the natural output, RECAST returns it as evidence rather than as an answer. It carries the IDs of the source records it was derived from, so any downstream consumer can trace the value back to its sources and inspect them with a simple lookup tool if needed. The AnswerLM then decides the final answer from this value together with the rest of the evidence and the task's requirements, for example converting units or following the required answer format, rather than returning the program output verbatim.

We then identify 3 key operational distinctions between agents that "constructing evidence" versus agents that "solving directly" and explain the importance of these designs.

Operational distinction:

  • State
    • Solving directly: The agent's state is its full interaction history: reasoning, tool calls, raw outputs and errors. The agent reads this history itself to produce the answer. This history does not need to be tidy or traceable, because the agent is its own audience.
    • Constructing evidence: The state is a size-limited evidence set that is handed to a separate AnswerLM. Only successful evidence is kept, and each evidence item must name the source records it came from to be accepted.
    • Importance: Evidence as state is more useful because 1) the answerer reasons over a compact, grounded input instead of a long, noisy history. Failed attempts and dead ends never reach it, and every fact can be traced to the source; 2) The constructed evidence is reusable. It could potentially be fed to different AnswerLMs or be used for other downstream tasks.
  • Terminal decision
    • Solving directly: The agent stops when it emits an answer.
    • Constructing evidence: The router stops with a judgment that the retained evidence is sufficient for a separate AnswerLM to answer correctly.
    • Importance: It is more challenging: the router must anticipate what a different model needs and construct the correct and necessary evidence for it.
  • Learning signal
    • Solving directly: The agent is trained on the correctness of its own answer.
    • Constructing evidence: The router is trained on the correctness of a frozen AnswerLM given the evidence it constructed. It is never trained to answer.
    • Importance: The router is credited only through a frozen AnswerLM's answer and cannot emit an answer itself. Its reward can therefore rise only through better evidence, not through guessing or memorizing answers, a shortcut that remains open to a direct-answer agent.

To summarize, constructing evidence changes what the agent keeps, when it stops, and what it is rewarded for: its output is a source-grounded evidence set that another model must be able to answer from, not an answer.

AuthorsOpenReview ↗

Response to N2Xk 2

N2Xk-Q1: "If an existing multi-turn tool agent were given the same retrieval, SQL, and Python interfaces, what would be RECAST's most essential mechanistic difference from it? Which of the reported results support the value of that difference?" W1(part1): "The boundary of the method's novelty relative to existing tool agents is not clearly established ... multi-turn tool execution, adjusting actions from execution feedback, and producing intermediate results through data transformations all have precedents..." W2(core): "The main comparisons do not separate the contribution of the method design from the contribution of tool and interaction capabilities."

In our response, we first clarify the contribution of RECAST, specify current and additional experiments needed to separate each contribution, and report the additional experiments and their conclusion.

The contribution of RECAST is threefold:

  • C1: Tool set expansion: we broaden evidence construction beyond retrieval, identify retrieval and computation as complementary means, and develop RECAST that supports adaptive tool routing that supports evidence constrution over heterogeneous sources
  • C2: Evidence-construction policy design: The RouterLM does more than selecting operations. In addition, it also formulates the operation and decides when the evidence suffices. It does this over an evidence state that the framework maintains and checks against source records, rather than over a free-form transcript.
  • C3: Learning: We apply SFT and GRPO to train the small RouterLM. Its reward comes from the frozen answerer's correctness.

We already have:

  • RECAST(SFT+GRPO): Success rate 75.6%; Contribution C1+C2+C3
  • RECAST(Training Free): Success rate 64.1%; Contribution C1+C2
  • RECAST(Training Free, Synthesis Only): Success rate 59.9%; Contribution C2
  • RECAST(Training Free, Primitive Only): Success rate 56.8%; Contribution C2

Thus, through existing experiments, we have shown that C2 < C1+C2 < C1+C2+C3

Now, to separate the contribution of the C2, we conduct additional experiments to show C1 < C1+C2 and C1+C3 < C1+C2+C3. For C1 (or C1+C3), it means a multi-turn tool use (trained) agent with same tools but no RECAST specific evidence construction policy design (as proposed by Q1)

We conducted the experiments and show the results in the table below:

  • E1 (Tool-list routing): a multi-turn tool-use agent with same base model, tools, compiler, answerer, framework-maintained evidence state as RECAST, with RECAST's RouterLM prompt replaced. The prompt states the goal and describes each tool and its call format. The router picks a tool and writes only its input (a query, SQL, or a free-text code request), without RECAST's structured formulation (evidence goal, required output, reason) or its guidance on choosing operations and stopping.
  • E2 (Search-R1[1]): Search-R1 agent with same base model, tools, compiler and answerer as RECAST. It interleaves free-form reasoning with tool calls in a single transcript of raw tool outputs, and at the end it writes the evidence for the answerer itself.
  • E3 (Search-R1 + GRPO): E2 trained with GRPO from raw Qwen3.5-9B, using the same training data, reward, steps, frozen answerer and judge as GRPO-only RECAST.
  • E4 (CodeAct[2]): CodeAct agent with the same base model, tools, compiler and answerer as RECAST. It writes and executes Python code actions over the source records in a multi-turn loop, with RECAST's tools available as Python functions, and revises from execution output. At the end it writes the evidence for the answerer itself.
MethodContributionDataBenchFinQAHiTabHotpotQALaMPMultiHierttAvg.
E1 Tool-list routingC178.034.020.065.071.044.052.0
E2 Search-R1C170.058.036.092.046.056.059.7
E3 Search-R1 (GRPO)C1+C383.059.050.092.056.063.067.2
E4 CodeActC167.055.037.094.047.053.058.8
RECAST(Training Free)C1+C286.057.060.783.068.329.364.1
RECAST(GRPO)C1+C2+C381.764.075.788.364.061.772.6

From the results, we can observe that

  • E1 Tool-list routing (52.0%) < RECAST(Training Free) (64.1%):
    • C1 < C1+C2
    • removing RECAST's formulation and decision guidance costs 12.1%. Selecting a tool and writing its input is not enough; formulating each operation and judging when the evidence suffices is what makes the tools effective.
  • E2 Search-R1 (59.7%) and E4 CodeAct (58.8%) < RECAST(Training Free) (64.1%) :
    • C1 < C1+C2
    • tools and multi-turn interaction alone don't explain RECAST's performance; how the evidence is built matters. The gap is concentrated where the evidence must be kept exact: HiTab (36.0/37.0 vs 60.7) and LaMP (46.0/47.0 vs 68.3).

---continues in next comment---

AuthorsOpenReview ↗

Response to N2Xk 3

---continues from previous comment---

  • E3 Search-R1 (GRPO) (67.2%) < RECAST(GRPO) (72.6%):
    • C1+C3 < C1+C2+C3
    • Training also doesn't close the gap, so the remaining difference is due to RECAST's design rather than to its training or tools.

Together, these comparisons isolate C2.

RECAST's contribution is therefore not any individual mechanism. It is a combination of (C1) an operation space spanning retrieval, relational queries and compiled code across heterogeneous sources, (C2) a policy that formulates each operation and judges sufficiency over a framework-maintained, source-checked evidence state, and (C3) training that policy to optimize evidence quality.

N2Xk-W2(specific points): "The retrieval baselines lack equivalent code computation, and Direct Code is restricted to single-shot generation without revision from execution feedback, while RECAST combines retrieval, computation, and multi-round feedback (Appendix C.1). The appendix itself notes that part of Direct Code's failures stem from output-interface and program-execution issues (Appendix E.1), and the operation-space ablations in Table 2 use untrained routers. The results therefore support RECAST being better than the evaluated configurations, but not an advantage of its specific routing and role division over existing multi-turn tool agents. The trained/untrained comparison shows that training helps, but does not resolve this attribution question."

In our response, we clarify the goal of existing operation-space ablations and conduct additional experiments to extend current retrieval baselines and Direct Code.

First, we would like to clarify that RECAST(Training Free, Synthesis Only) in Table 2 retains multi-round execution feedback. This variant and the full training-free RECAST use the same untrained router, making their comparison matched with respect to training. Their results (59.9% versus 64.1%) support the complementarity of primitive and synthesized operations within the training-free framework.

Since this ablation retains RECAST’s architecture, we perform additional experiments including 1) multi-turn Direct Code, 2) existing multi-turn code-action agent CodeAct, and 3) tool-matched agent to better compare with RECAST.

Specifically, we

  • extend Direct Code to multi-turn execution, allowing it to inspect execution results and errors and revise its program
  • supply CodeAct[2] agent, which writes and executes Python code actions over the source records in a multi-turn loop, with the same base model, tools, compiler and answerer as RECAST. (as presented as E4 in previous question's answer)
  • augment IRCoT and Interact-RAG, the strongest iterative and agentic retrieval baselines, with same capabilities as RECAST including retrieval, SQL (new), python code generation with same frozen compiler(new), execution feedback, and up to six rounds.
MethodDataBenchFinQAHiTabHotpotQALaMPMultiHiertTAvg.
Multi-turn Direct Code87.08.031.066.064.04.043.3
CodeAct67.055.037.094.047.053.058.8
Tool-augmented IRCoT85.029.028.072.070.058.057.0
Tool-augmented Interact-RAG76.041.048.059.065.050.056.5
RECAST (Training-Free)86.057.060.783.068.329.364.1

From the results, with same Qwen 3.5 9B as the LLM, multi-turn Direct Code achieves 43.3%, CodeAct achieves 58.8%, and the augmented IRCoT and Interact-RAG achieve 57.0% and 56.5%, respectively. Training-free RECAST achieves 64.1%, still outperforms all previous baselines. These results indicate that the performance gap is not explained solely by the original baselines’ lack of computational tools or multi-turn execution feedback.

This affirms the observation and conclusion in the previous answer: tools and multi-turn interaction alone don't explain RECAST's performance. Formulating each operation, judging when the evidence suffices, and how the evidence is built matters.

In addition to tools and multi-turn interaction, we further validate the role division in N2Xk-Q4. We provide details of Direct Code's failure analysis in N2Xk-Q3.

N2Xk-Q3: "How much of Direct Code's failure rate comes from interface compliance or execution errors, and how much from task-reasoning errors? Does the qualitative analysis in Appendix E.1 imply that its lower scores partly reflect the single-shot setup"

We compared two runs of the single-shot Direct Code and multi-turn Direct Code using Qwen3.5-9B on the same 600 questions.

Final outcomeSingle-shotMulti-turn
Correct answer38.0%43.3%
Interface or execution failure45.8%20.2%
Successful execution, incorrect answer16.2%36.5%

---continues in next comment---

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.