- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces BVEd, a parametric experience-memory framework that transfers information from past reasoning attempts to unseen problems. The method collects successful and failed attempts from a frozen solver, uses a teacher model to extract candidate lessons, and applies source-side screening and benchmark-specific selection strategies to construct training targets. These lessons are distilled into a helper model that generates problem-specific guidance for new questions without retrieving past trajectories or modifying the solver. Experiments on four deductive-reasoning benchmarks, including FOLIO, FLD×2, ProofWriter, and RuleTaker, report accuracy improvements of 3.09–14.87 percentage points over direct answering.
Critical strengths
S.1. Well-motivated research problem. The paper addresses an important question of how information accumulated through repeated reasoning attempts can be reused on unseen problems. The distinction between solving an individual problem and consolidating past experience into reusable guidance provides a meaningful motivation for the study.
S.2. Positive empirical results across multiple benchmarks. BVEd achieves positive accuracy gains across all four reasoning benchmarks under the evaluated protocols, with relatively small variation across helper training runs. The use of paired evaluations and additional uncertainty analysis on FOLIO strengthens the empirical presentation of these results.
S.3. Informative analysis beyond aggregate accuracy. The paper examines how guidance changes individual predictions through repair–harm analysis and evaluates its effects across different recipient models. These analyses provide a more detailed characterization of the guidance's effects on individual decisions and its variation across recipient models.
Critical weaknesses
W.1: Insufficient ablations of experience extraction and screening.
The paper does not isolate the contributions of success/failure histories or individual screening components (ABG, blind-JSD, and lexical filtering). Existing controls demonstrate the value of problem-specific guidance but not the necessity of the proposed experience-learning pipeline. Comparisons with success-only histories, history-free guidance distillation, and unfiltered lessons under matched training conditions are needed.
W.2: Insufficient comparison with standard reasoning baselines.
The main experiments compare answer-only Qwen3-8B with BVEd, which introduces an additional trained 4B helper. However, short-reasoning and zero-shot CoT baselines using the same Qwen3-8B are missing. Appendix E provides limited reasoning controls on smaller solvers. Comparisons under controlled reasoning lengths and end-to-end inference costs are needed to establish benefits beyond additional inference computation.
W.3: Insufficient analysis of answer-equivalent test-time guidance.
Although BVEd screens training lessons, Appendices C and G.1 show that test-time guidance can still contain answer-equivalent conclusions after postprocessing. The paper does not quantify how frequently this occurs or how much it contributes to accuracy gains. Without such analysis, the role of reusable reasoning guidance versus direct answer provision by the helper remains unclear.
Questions for authors
Q1. How much do experience histories and individual screening components contribute compared with success-only, history-free, and unfiltered guidance under matched training conditions?
Q2. Does BVEd outperform pretrained-helper, short-reasoning, zero-shot CoT, and training-free methods (e.g., Self-Consistency and Self-Refine) under comparable inference costs on the main Qwen3-8B benchmarks?
Q3. How frequently does test-time guidance contain answer-equivalent conclusions after postprocessing, and how much of BVEd's accuracy gain remains when such conclusions are removed?
Minor Comment: Presentation and Visualization Quality The overview figure (Figure 1) is overly crowded, with small text that reduces readability. Figure 3 on page 8 also contains excessive whitespace, thin bars, and small labels. A more compact layout and clearer visual design would improve the overall presentation.
Thank you for your detailed response and for carrying out the extra experiments. I appreciate the authors' efforts to address my concerns, and I have no further questions at this stage. In my opinion, it was unnecessary to repeatedly emphasize the 95% confidence intervals, as they were not central to my main concerns. I still feel that the paper's methodology and empirical evidence could be strengthened further. As shown in Table R2, the proposed screening strategy offers no clear advantage over random selection. Therefore, I will maintain my original rating.