- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Reader-Oriented Evidence Reshaping (ROER), a framework for optimizing retrieved evidence for a frozen language-model reader. It combines multi-view retrieval with executable Evidence Organization Programs (EOPs), which compose operators for evidence selection and organization. Candidate programs are evolved offline using reader-derived utility, including proposals informed by execution failures. A learned selector then chooses one program per query, requiring one EOP execution and one reader call at inference time.
Experiments evaluate 200 examples each from CosmosQA, QuALITY, and LongBench-v2. With Qwen3-8B, ROER achieves the highest average accuracy and the best results on two benchmarks. Its largest improvement is on LongBench-v2: 40% versus 33% for the strongest reported baseline. Additional experiments evaluate two other readers, component ablations, and inference latency.
Critical strengths
-
A practical design for adapting evidence processing while keeping the reader frozen. The constrained program space makes the transformations explicit, while offline search and a lightweight selector avoid searching over programs at inference time. This is a useful engineering contribution: it offers an interpretable way to customize evidence processing without updating the reader’s parameters.
-
Encouraging long-context results across several readers. The reported LongBench-v2 improvement from 33% to 40% with Qwen3-8B is meaningful enough to warrant investigation. ROER also outperforms the evaluated baselines with the other two readers, and removing the reshaping module reduces accuracy from 40% to 30.5%. These results support the usefulness of the overall system, subject to the protocol and attribution concerns below.
Critical weaknesses
-
The conceptual novelty is overstated, and the specific increment over existing work is not sufficiently established.
Optimizing retrieved context for a downstream reader is an established research direction. RECOMP trains compressors using downstream language-model performance, and PRCA explicitly uses a frozen generator’s feedback to optimize an adapter between retrieval and generation. FaviComp, which this paper evaluates, already adapts compressed evidence to the target model’s familiarity. Evolution guided by execution traces also has a clear precedent in GEPA.
Against this background, ROER’s potentially distinctive contribution is its compositional EOP search space and query-dependent program selection. However, the experiments do not separate the benefits of evolution from those of the eight manually designed initial programs, or adaptive selection from simply using the best fixed program. Furthermore, EOPs change evidence content and coverage as well as presentation, so the reported gains cannot yet be attributed specifically to alignment with a reader’s representation preferences.
Theorem 1 does not substantially strengthen this argument. Its condition is algebraically equivalent to . The result is correct, but it restates the existence of a better representation under the second reader rather than explaining when such a preference reversal should occur or why ROER will discover it.
-
The evaluation leaves important uncertainty about independence, comparability, and robustness.
Appendix C.1 states that each dataset’s original public training split is used for optimization and its public validation split for evaluation. However, the official LongBench-v2 release contains one split named
train, comprising 503 benchmark examples, rather than separate public training and validation splits. A custom split could be legitimate, but the actual partition and its independence from EOP evolution, failure-trace collection, and selector tuning need to be documented before the main result can be assessed confidently.The comparison protocol also needs clarification. ROER limits both intermediate evidence packing and final reader evidence to 8,000 characters, while corresponding evidence access, output budgets, model settings, and timing boundaries for the baselines are not sufficiently documented. Consequently, the accuracy and latency tables do not isolate the effect of evidence organization.
Finally, each benchmark uses only 200 evaluation examples, without uncertainty estimates or repeated sampling. The headline LongBench-v2 improvement corresponds to 14 additional correct answers; the CosmosQA improvement over the strongest baseline corresponds to three. These gains may be real, but paired uncertainty estimates are needed to assess their robustness. On QuALITY, ROER trails the strongest baseline, so the claims should also distinguish improved average performance from improvement on every benchmark.
Questions for authors
-
Evaluation protocol clarification: What exact LongBench-v2 examples were used for EOP evolution, selector training and validation, and final evaluation? Please provide the split construction, seeds, and example IDs, and clarify whether any evaluation examples or their labels influenced program proposals, failure traces, or tuning.
-
Evidence needed to assess the claimed increment: With the retrieved evidence pool and output budget held fixed, how does ROER compare with the best fixed EOP and a selector trained only over the eight initial EOPs? Does transferring an EOP bank and selector between readers measurably reduce performance relative to optimizing them for the destination reader? These comparisons would directly affect my assessment of whether evolution and reader-specific adaptation provide distinct benefits.
-
Comparison and robustness clarification: What evidence inputs, length budgets, exact model checkpoints, and decoding settings were used for each baseline? What stages are included in the latency measurements? Please also provide paired uncertainty estimates for the main accuracy differences and report the offline reader-call and proposal-model costs.