- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper introduces ReSolve, an inference method that uses the visible candidate reasoning to generate a repaired answer for problems whose multi-sample answers do not agree with each other or have no parseable answer. Experiments on 130 math problems with Qwen3-4B-Thinking-2507 show that ReSolve can correct several answers but is not significantly better than SC@8
Critical strengths
- The control experiment design is careful and clean. The key same-pool controls share the initial four candidates, and the answer-only ablation removes only the visible derivations.
Critical weaknesses
- The paper is hard to read. Something is extremely strange in the paper: (1) the symbols of F/H/D have no explanations. It is unusually to use "F" as a symbol for experimental setting (and "F" is started at "F2", not "F1"). (2) The only significant comparison illustrating ReSolve's clear effectiveness advantage is against SC@4 (but SC@4 has a lower budget cost). According to Appendix B, this comparison was added after the initial evaluation. I think the defensive writing in Appendix is more like an AI transparency disclosure. Thus, I think this paper is not ready for publish and needs more human inspections and major revisions before acceptance.
- ReSolve does not show a clear advantage over baselines in Table2, and all methods here are nearly saturated (close to Oracle's performance). I understand this might due to the dataset difficulty. However, such minor difference can be triggered by either randomness or cherrypick (since there are F2 and F3 settings but not F1, I am concerned more about this). Thus, I believe a repeated sampling evaluation or using more difficult datasets with clear discriminative capability are needed (I know that "repeated sampling seems like an AI generated comment, but I do believe it is needed in this paper).
- LLM-Blender and MoA have already provide the method paradigm like "generating a new answer from existing candidates". The paper cites these works in Section 4 but neither compare with any of them nor explain the reason why not to compare with them.
- The only new component in this paper method, i.e., the specialized route instructions, do not show clearly better than Uniform.
- This submission does not appear to provide an artifact package with reproducible code. Thus, I am not convinced by existing results.
Questions for authors
- What do F, H and D stand for, and is there an F1? Can you add a table in the main text summarizing F2/F3/H1/D1?
- Does ReSolve really show a stable performance over existing methods like MoA and LLM-Blender? If you think they are not fair baselines, please provide explanations. If not, it is better to provide an extra experiment. I do not expect ReSolve to be the most effective one, but a pareto optimization analysis is needed.
- Can you provide a repeated sampling experiment to illustrate the stability of ReSolve's advantage? (please use fixed seeds). Also, can you compare ReSolve with sampling more candidates under the same token budget on the triggered problems only? (Again, this is not AI-generated comment. I just need more convincing evidences.)
- If you choose to provide the experiments above, please also release the reproduction artifacts like code and configs. I am interested to reproduce these cost-efficiency-related works.
2 comments
Response to Reviewer Dexr [2/2] — Setting names, F1, repeated sampling, and artifacts
Setting names are now explicit in the main text. Table 2 (p. 4) maps Competition Pool A/B to the original F2/F3 identifiers and lists the questions, model/budget and purpose of the historical studies. F means fresh candidate sampling; H1 is the historical shared pool; D1 is the diagnostic intervention study on that same pool. A/B each contain 130 competition questions; H1/D1 share 160 questions and a separate 16K moderator-output protocol. Main prose uses the descriptive names, retaining the original IDs for traceability.
F1 exists and is reported. It is the earlier completed candidate draw, superseded in the original primary source registry by F2/F3. We matched the archived F1 summary to its source manifest and added the completed-draw inventory in Table 25 (p. 25):
| Completed draw | SC4 | SC8 | ReSolve | ReSolve / SC8 tokens M |
|---|---|---|---|---|
| Earlier F1 | 90 | 91 | 94 | 12.257 / 23.140 |
| Pool A (F2) | 91 | 94 | 100 | 12.464 / 23.193 |
| Pool B (F3) | 92 | 96 | 99 | 12.349 / 23.378 |
All three completed draws show more solved questions than SC4/SC8 and approximately 46–47% fewer tokens than SC8. A/B remain the primary pools for the matched mechanism interventions. The appendix records fixed sampling identities and candidate ordering; repeated outcomes remain grouped by the same 130 problem identities rather than treated as new questions.
The post-hoc SC4 comparison. Its secondary status is retained. The new positive argument directly answers the requested budget question: same-gate continuation under comparable budget yields 91/92 versus ReSolve's 100/99, and the full-length curve yields 94/96 at greater cost. Table R1 in our response to Reviewer Ho3k [1/2] reports every intermediate point and the exact paired tests. This makes the continued-sampling controls, expanded evaluation and actual cost frontier central to the revision.
Reproduction materials. We prepared an anonymous source bundle containing frozen inference/controller code, sampling configurations, hash-checked score summaries, paired records and CPU table reconstruction. A portable inference entry point and controller-parity checks accompany the source; CPU table reconstruction requires no model weights or API credentials. The revision's reproduction attachment is the access point for this bundle. Appendix J (p. 25) explains the inventory and reconstruction procedure.
The shared results are centralized to reduce repetition: our response to Reviewer Dexr [1/2] contains fusion and Pareto comparisons, our response to Reviewer Ho3k [1/2] contains selective sampling, and our response to Reviewer Whwx [2/2] contains the full external benchmark and two new backbones. Each corresponds to an actual table or figure in the revised PDF.
Response to Reviewer Dexr [1/2] — Fusion comparison, Pareto analysis, and contribution
We added the requested direct generative-fusion control and cost–accuracy analysis. The aggregator uses the same model and four candidate solutions as ReSolve, with an independent neutral synthesis instruction. We evaluate one aggregation layer both on every question and under the original invocation gate.
Table R4. Same-model fusion and moderation. Entries are Hybrid correct / total tokens M on 130 questions per pool.
Relative to always-on fusion, ReSolve uses 6.63%/7.12% fewer tokens, with solved counts 100/99 versus 101/98. The gated fusion and one-call points are explicitly included in the revised Table 6 (p. 7) and Figure 2 (p. 6). Readers can inspect accuracy and cost jointly rather than infer a frontier from one comparator.
Relation to MoA and LLM-Blender. MoA motivates aggregation from candidate responses; our controlled adaptation fixes the proposer, aggregator weights, candidate evidence and accounting to isolate the aggregation operation. It is labeled same-model, single-layer MoA-style fusion. LLM-Blender additionally uses separately trained PairRanker and GenFuser components. The revised related work and Appendix H.3 (p. 21) state these component differences and distinguish the controlled adaptation from a full heterogeneous MoA or LLM-Blender reproduction.
What the additional evidence contributes. ReSolve defines a bounded policy linking answer availability, evidence-conditioned moderation and acceptance. Its mechanism is now tested against independent restart under the same gate, rather than only fixed SC@8. Table R1 in our response to Reviewer Ho3k [1/2] provides all budget-matched and full-length sampling points: ReSolve solves 100/99 versus 94/96 for same-gate +4, using 23.47%/25.09% fewer tokens. This directly addresses the possibility that savings arise only from selective allocation.
The evidence intervention is also replicated: deleting derivations changes 100→93 and 99→95 while increasing moderation cost. Response to Reviewer Whwx [1/2] gives the complete intervention and paired results. Table R3 in our response to Reviewer Whwx [2/2] adds 674 OlympiadBench questions and two backbones, where ReSolve again achieves lower token consumption than SC@8 with equal or higher solved counts. The revised Introduction organizes the contribution around these experimentally separated decisions.
Beyond selection. A fixed-pool oracle uses the reference answer to choose among existing candidates. ReSolve instead generates from candidate evidence without reference-answer access. The Olympiad extension includes two Oracle@4-unsolved recoveries (indices 109/395); the revised case study and appendix trace how unfinished intermediate work supports a new answer.