- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes FORGE, a framework that dynamically selects the form of external evidence provided to a frozen LLM when answering knowledge-intensive questions. The framework routes each query among predefined support forms, including Direct, Summary, and Raw, while also considering different reasoning budgets, with the goal of maximizing answer quality while minimizing token cost.
The router is trained through KL distillation from a utility-based Boltzmann distribution, followed by GRPO-based refinement using feedback from the frozen host model. Experiments are conducted on five knowledge-intensive QA and fact-verification benchmarks across eight frozen LLM backbones. The results demonstrate improvements in QA accuracy and token efficiency compared with fixed inference configurations.
Critical strengths
-
This paper addresses a practically valuable problem: how to maximize QA accuracy and minimize token cost by dynamically switching between several RAG configurations. It also proposes a "utility calculation method." The proposed framework is practical by offering a lightweight mechanism, and the empirical results indicate that query-dependent routing can improve the quality-cost tradeoff compared with fixed inference configurations.
-
The evaluation covers a broad range of model families by testing eight frozen LLM backbones, ranging from open-weight models to large API-served models. This demonstrates FORGE's transferability and generalization capabilities. The authors also provide ablations separating action-space expansion from GRPO refinement, cross-host transfer experiments, and measurements of fresh-online inference costs. These analyses strengthen the empirical evaluation and help clarify the conditions under which the method is practically useful.
Critical weaknesses
- Limited scientific insight into the underlying routing problem and insufficient methodological novelty.
The central contribution of FORGE is a lightweight router that maps query features to predefined combinations of evidence support forms and reasoning configurations. While this is a reasonable engineering approach, the paper provides limited insight into the fundamental question motivating the method: what properties of a query determine whether external evidence is necessary, whether compressed evidence is sufficient, and how much reasoning is required?
Although a router is obtained through KL distillation and reinforcement learning exploration, the paper provides no insight into why a particular question is better answered directly by the model, with reference to raw text, or with reference to a summary. This may be the scientific question worth exploring. Otherwise, the method appears to be a classifier that distinguishes between knowledge already possessed by the current model, knowledge that can be summarized from the dataset, and knowledge contained in the raw text of the database.
- Narrow task generality and limited evidence for broader agent adaptation.
Although the paper evaluates five benchmarks, they all belong to a relatively narrow family of knowledge-intensive question answering and fact verification. The method operates as a single-step, pre-answer router over three predefined evidence forms, rather than performing adaptive retrieval, iterative search, or sequential agent decision-making.
Although the paper focuses on QA-type benchmarks, today's QA benchmarks have also been widely synthesized and extended, for example, to involve multi-hop reasoning or parallel exploration of multiple questions. Despite conducting many ablation experiments, the paper falls short of proposing a method that can be more broadly applied to QA-type tasks.
Questions for authors
Q1. What generalizable query properties does FORGE actually learn?
Q2. What is the specific advantage of joint evidence and reasoning optimization?