- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Verified Cognitive Schema Transfer (VCST), a training-free framework that stores and reuses problem-solving knowledge across a stream of tasks at test time. When a candidate solution passes a verifier, an LLM summarizes it into a structured schema entry and then store it in a persistent memory. For each new task, VCST retrieves schemas through tag-based Jacquard filtering and uses the LLM to check their applicability and rerank them, keeping at most two. It then includes the retrieved schemas in the generation prompt to produce candidate solutions. Under a fixed API budget, the LLM also decides at each step whether to solve a task, synthesize provisional schemas, or consolidate existing ones. Experiments on LiveCodeBench, multi-task reasoning, code reasoning, and multi-language code generation show that VCST outperforms test-time scaling and memory-based baselines.
Critical strengths
- The evaluation is broad, covering 13 benchmarks across four task groups and nine backbones from three model families (GPT, Qwen2.5-Coder, and DeepSeek-R1-Distill) and scales on LiveCodeBench.
Critical weaknesses
- The core idea of distilling successful solutions into reusable memory and retrieving them for later tasks has been explored extensively, but several closely related works are not cited. Buffer of Thoughts retrieves distilled thought templates for new problems, and Agent Workflow Memory induces reusable workflows from successful trajectories in an online setting. Dynamic Cheatsheet addresses essentially the same problem of test-time learning with an adaptive memory over a stream of tasks. Given this prior work, the contribution appears to reduce to verified-only admission and an applicability field in each memory entry. The authors should discuss these works, clarify what is specific to VCST, and include Dynamic Cheatsheet as a baseline.
- For benchmarks without an executable verifier, such as MATH and MMLU-CF, the correctness feedback can only come from ground-truth answers, a stronger assumption than label-free memory methods. The memory baselines were also originally designed for agentic or conversational settings, and ReasoningBank relies on an LLM judge rather than labels. The paper should state the feedback source for each benchmark, describe how each baseline was adapted and what feedback it received, and report a variant of VCST without correctness feedback.
Questions for authors
- On LiveCodeBench, does the verifier V use only the public tests, or does it overlap with the hidden tests used for scoring?
- The large gains over S* are reported in Figure 3, where VCST repairs failed candidates and revisits unsolved tasks. However, API calls per problem are reported only for the single-visit setting in Table 1, and only a lower bound of 16 calls is given for S*. How many model calls per problem do VCST and S* use in the Figure 3 setting, and what value of the budget B is used?
3 comments
Thank you for the concise response and the additional experiments. The results now clearly show that your method achieves better performance across various tasks and models, as well as better efficiency than S*. Still, I find the methodological novelty somewhat limited. Revisiting can only be applied to contest-style settings. For the dynamic allocation, Table 4 shows a modest gain compared to a fixed policy. Since most of my concerns have been addressed, I have raised my rating from 2 to 3.
Response to Reviewer GCrG
Thank you for your updated assessment. We are pleased that the new experiments have addressed most of your concerns and made the performance and efficiency benefits clearer. The specific points raised in your review guided several of these additions and helped us improve the paper. We appreciate your reconsideration and constructive suggestions.
Response to Reviewer GCrG
We thank the reviewer for the constructive comments.
1. Related work and contribution. We now cite and discuss Buffer of Thoughts, Agent Workflow Memory and Dynamic Cheatsheet (Section 2.2).
VCST asks how inference-time knowledge can advance unfinished tasks within a shared budget. In full code-solving, VCST starts with empty memory and revisits unresolved tasks after other tasks yield or improve schemas: later experience can help complete earlier unfinished problems.
Knowledge construction is itself a choice about where to spend inference computation. Using task state S and knowledge state K, VCST allocates budget B between searching for particular answers and building knowledge useful across tasks, through solve, explore and optimize. Round-robin orders task visits; adaptation selects operations (Sections 3.1–3.3; Appendix G, Algorithm 1).
Schemas abstract feedback-supported experience into shared strategies and parameterized templates, with provenance and applicability conditions compared with new task constraints. Abstraction provides transferable solving structures; applicability assessment determines where to reuse them.
We add ablations of abstraction (Full 88.3%, task-level-only 85.0%) and applicability (semantic-only 86.3%; Table 4), plus same-pool retrieval and paired-removal tests of reuse (Appendix H).
2. Dynamic Cheatsheet baseline. We add a matched comparison with task-adapted Dynamic Cheatsheet-Cu, retaining its released curator and full-cheatsheet context. Both methods use GPT-5-nano, the same 240-task LiveCodeBench order, empty initial memory, public feedback and solver: five candidates/visit, up to three repair rounds for the top two, at most two visits/task.
VCST solves 212/240 (88.3%), versus 206/240 (85.8%) for DC-Cu: six more tasks, or 2.5 percentage points. This tests the complete memory systems under matched feedback (Section 4.5; J.2, Table 13).
3. Feedback assumptions and adaptation. We agree that reference grading is a stronger deployment assumption. In Table 2, MATH/GSM8K/BBH/MMLU-CF use reference grading, MBPP/HumanEval+ execution checks, and LongBench reference-based metrics. Other code tasks use execution checks; HaltEval uses halting labels. In single-submission runs, feedback arrives after answers are fixed and updates subsequent-task memory only (Section 4.2; Appendix F).
ExpeL, A-MEM and ReasoningBank all receive post-submission benchmark feedback: task, own answer, success/failure and feedback text. They retain insight extraction, linked-note updates and success/failure reflection, respectively; VCST admits source experiences judged successful.
We add a method × feedback experiment comparing VCST and ReasoningBank on MATH/GSM8K/BBH/MMLU-CF with GPT-4o-mini, to test use without external labels. Conditions share generation settings, start with independent empty memories and receive either reference-feedback bits or bits from a shared judge without reference answers. Judge rationales are excluded from updates; under self-judging, reference scoring only evaluates frozen answers. VCST's mean falls 85.48→83.38 (−2.10 points), but it leads ReasoningBank by 1.68 points with reference feedback and 1.78 with self-judging. Thus, the gain persists without external correctness labels in this control, reported separately from Table 2 (Section 4.5, Figure 5; J.1, Table 12).
4. Public versus hidden tests. We clarify: LiveCodeBench uses public tests for ranking, repair and memory admission. Hidden tests score frozen final programs only; their contents/results never enter generation, repair, retrieval, schema construction, action selection, revisits or stopping. Both sets test the same problems (Section 4.1; H.2/J.2). Appendix E/Table 9 separately reports public-test diagnostics.
5. Full-process costs and budget. We add full-process costs for all 511 tasks and nine backbones, including memory overhead and revisits (Appendix I, Figure 7). Figure 3's cap is B=20,000 model calls/run; VCST generates five candidates/visit, repairs the top two for up to three rounds, and allows at most three visits/task.
All generation, repair, retrieval, schema construction/maintenance, action-selection and retry calls count; tokens include input/output and supplied memory. Totals are divided by all 511 tasks, including failures; revisits add consumption, not tasks. VCST uses 31.4–51.0% fewer calls and 39.2–51.4% fewer tokens across paired backbones. Table 1's 5.8–7.9 calls/task covers one visit without repair/revisits; S*'s 16 samples likewise exclude debugging. These are realized costs.