- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces REPROT, which searches over typed reasoning protocols specifying operations, visibility of intermediate artifacts, control flow, and answer commitment. A synthesizer proposes patches from labeled search-set execution digests, including state-conditioned computations and bounded instruction changes. Patches are screened using paired fix/break counts, a one-sided sign-test threshold, and a mean-call budget. Final selection uses pooled search and validation performance before freezing the protocol for testing. Experiments use one executor backbone on eight benchmarks, comparing seventeen reproduced baselines and the original ADAS and AFlow implementations. REPROT reports the highest average accuracy, 84.18%, at 3.9 calls per question.
Critical strengths
- Broad evaluation with meaningful aggregate improvements. The shared-backbone comparison covers eight tasks and includes direct runs of ADAS and AFlow, rather than relying only on topology baselines. REPROT averages 84.18% versus AFlow's 82.56%, and offers a favorable accuracy/call trade-off relative to the reproduced topology methods. The appendix also reports token usage and offline search cost. This makes the method practically interesting, while recognizing that AFlow remains cheaper online at 3.2 calls versus 3.9.
- An auditable representation and unusually explicit accounting of learned changes. Typed artifacts, enforced visibility, static validation, and commitment rules provide a clear implementation framework. The appendix records accepted patches, their fix/break counts, instructions, selection outcomes, and both successful and unsuccessful cases. The authors explicitly identify format-related gains on TheoremQA, instruction-only learning on MuSR, and selection of an unchanged seed on MMLU-Pro. The story-disjoint MuSR split is also an appropriate safeguard.
Critical weaknesses
- The distinctive contribution beyond existing workflow and prompt optimization is not adequately isolated. The introduction characterizes prior ADAS/AFlow updates as unconditional, but AFlow explicitly represents conditional logic and provides test-dependent correction workflows (Figure 3 and Appendix A.5 of Zhang et al., ICLR 2025). The plausible distinction is the constrained representation and gated patch-search procedure, rather than the ability to represent conditional computation. Yet structure-only and free-code controls differ in pools, rounds, digests, and, for free code, final selection rules; Appendix I also documents a dead branch caused by an await issue. The largest gains come from instructions, and MMLU-Pro returns an unchanged seed. Furthermore, the 9.7-point LiveCodeBench first-branch loss disables the cascade seed's existing escalation, so it does not isolate learned trigger improvement. These results demonstrate a useful overall system but do not yet establish the specific benefit of learning protocol structure over matched prompt optimization and seed selection.
- Search-time sampling semantics and the unit of statistical analysis need clarification. Appendix C.6 describes caching by model, prompt, and temperature, while Appendix D specifies identical prompts and temperatures across self-consistency branches. The cascade's initial solvers also both appear to use temperature 0 under the documented defaults. If identical requests share a cached response within a run, independent sampling and disagreement-triggered search behavior would be compromised. The actual implementation may preserve separate sample slots, but this is not explained and matters for the reported branch-dependent patches. Separately, sign tests pool three seed/item outcomes for each question, without retaining question-level dependence or story-level dependence in MuSR. The repeated evaluations of frozen REPROT protocols also do not measure search variability. These concerns warrant clarification and clustered reanalysis; they do not establish a test-cache bug or test-set leakage.
Questions for authors
- Does the cache key or lookup procedure include a node, sample slot, invocation index, or another distinction beyond the stated model/prompt/temperature tuple? Please explain how identical SC branches and the two initial cascade solvers retain independent samples within a run while preserving unchanged-prefix pairing across patches. A minimal search-time trace or implementation excerpt would be particularly useful.
- On a representative branching task and an instruction-dominated task, can you compare instruction-only optimization of a fixed seed with full search under the final pools, digests, rounds, synthesizer, gate, and selection rule? For LiveCodeBench, can you preserve the seed's cascade conditions and remove only the learned modification, reporting paired fixes/breaks and cost?
- How should the claimed contrast with unconditional prior workflows be revised given AFlow's conditional code examples? Can the free-code comparison use matched selection rules and a corrected implementation, or can its evidential role be explicitly limited?
- Do the AFlow advantage and per-benchmark conclusions persist under question-clustered resampling, with story-level clustering for MuSR? Please distinguish search seeds from execution seeds and clarify that the repeatedly reused p≤0.2 search gate is a heuristic rather than a calibrated discovery or generalization guarantee.
8 comments
[W1b & Q2] Isolating Learned Structure from Instructions and Seed Selection [part 1/2]
Thank you for these controls. Our answer to Q2: we ran instruction-only optimization under the final pools, digests, rounds, synthesizer, gate and selection rule on all eight benchmarks. On the instruction-dominated tasks (TheoremQA, MuSR), full search performs similarly to instruction-only search (83.5 vs. 83.2; 79.1 vs. 78.6), and the paper now calls these instruction gains. On LiveCodeBench, keeping the seed's cascade condition and removing only the learned modification costs 4.6 points (34 fixes vs. 10 breaks) for 0.1 mean calls. Beyond LiveCodeBench the structural contribution is positive but smaller, and two parts of the mechanism are not isolated; part 2/2 lists them.
1. Instruction-only optimization under the final setup (§5.3, Table 2a, App. K.1, Table 12; three seeds):
| MATH | Olymp | ThQA | LCB | GPQA | SGPQA | MMLU-P | MuSR | Avg | Calls | |
|---|---|---|---|---|---|---|---|---|---|---|
| Seed selection only | 95.7 | 86.0 | 72.9 | 86.5 | 86.4 | 62.1 | 86.2 | 60.3 | 79.50 | 4.5 |
| Structure only | 95.6 | 86.6 | 73.5 | 86.3 | 86.7 | 62.5 | 86.2 | 61.4 | 79.86 | 4.3 |
| Instruction only | 95.7 | 87.8 | 83.2 | 86.1 | 86.4 | 62.2 | 86.2 | 78.6 | 83.26 | 3.4 |
| Full | 95.9 | 88.9 | 83.5 | 89.0 | 87.7 | 63.1 | 86.2 | 79.1 | 84.17 | 3.9 |
- Instruction-dominated tasks (TheoremQA, MuSR): §5.3 now says: "Instruction-only search is within 0.5 points of the full search on TheoremQA and MuSR, whose learned changes are single instructions (answer format; story tracking)."
- Overall: full − instruction-only is +0.90 [0.13, 1.69], and +1.3 [0.2, 2.3] on the five benchmarks whose selected programs contain structural patches.
- LCB alone: +2.9 [−0.6, 6.3], not significant against this arm; point 2 gives the cleaner LCB comparison.
- MMLU-Pro: all four arms return the unpatched Debate seed. Its gain over the PHP root is a selection effect, not a structural one.
- The instruction-only arm is not structure-free: it inherits hand-designed structure through the seeds it selects (Eq. 19); part 2/2 removes the library.
2. Q2, LiveCodeBench with the seed's cascade condition kept (§5.3, App. K.8, Table 18). You rightly note that the 9.7-point first-branch loss also disables the cascade's own escalation. We now keep the seed and its agree(c1, c2) test and vary only the learned critique:
| LCB (3 seeds) | Acc | Calls |
|---|---|---|
| : cascade seed, learned patch removed | 84.4 ± 1.2 | 2.7 |
| : critique in the disagreement branch | 89.0 ± 2.0 | 2.9 |
| same trigger, one generic extra solve and a six-way vote | 85.7 | 2.9 |
| critique on every item (with d1–d3, v1, v2 run everywhere) | 87.8 ± 2.0 | 8.0 |
| , c2 at | 88.6 ± 1.7 | 3.3 |
- Paired fixes and breaks: versus gives 34 fixes and 10 breaks pooled over three seeds; the question-clustered difference is +4.6 [1.4, 7.9].
- Cost: +0.15 mean calls: the critique runs on the 77 of 525 outcomes where the first two programs disagree.
- What LCB learned, and what it did not. The search learned the critique, its reads (programs and verifier evidence, not full reasoning) and its place in the seed's existing disagreement branch. It did not learn the condition; that is the seed's.
- Under the same cascade trigger and mean calls, replacing the learned critique with a generic extra solve reduces accuracy by 3.2 points (95% question-clustered CI [0.2, 6.3]; 467 vs. 450 of 525). This supports the learned sub-protocol in this setting; it does not establish a general advantage over self-consistency or newly learned trigger predicates. Calls are matched, tokens are not.
- Run everywhere, the critique costs 2.8× the calls; on items where the first two programs agree it fixes 11 and breaks 17.
- The submitted override stays in Table 11 with your caveat (App. H.2); it is no longer our evidence.
3. Difficulty (Table 20). The critique fires on 2%, 10% and 36% of easy, medium and hard LCB items.
4. The same instructions with at least as many calls (§5.3, Table 2b, Table 14). Self-consistency receives ReProt's learned instructions and samples per benchmark:
| + ReProt's instructions | Avg | Calls | ReProt − method |
|---|---|---|---|
| Self-consistency, | 83.35 | 4.3 | +0.82 [0.08, 1.57] |
On LCB the learned instruction belongs to the critique node, which self-consistency lacks, so this control is not fully matched; without LCB the difference is +0.42 [−0.05, 0.90]. On TheoremQA () and MuSR, the self-consistency program is itself. We read the control as a bound: with the same instructions and at least as many calls, plain sampling comes within 0.82 points of ReProt on average, a difference significant only with LCB included. On LCB alone it is +3.6 [−1.3, 8.6], not significant; the call-matched LCB evidence is the generic-solve row of point 2.
[W1b & Q2] Isolating Learned Structure from Instructions and Seed Selection [part 2/2]
5. Removing the hand-designed library (§5.3, Table 2a, App. K.2, Table 13). Root and archive contain only s1 = solve[cot](problem); commit s1.
| From one CoT call | MATH | Olymp | ThQA | LCB | GPQA | SGPQA | MMLU-P | MuSR | Avg | Calls |
|---|---|---|---|---|---|---|---|---|---|---|
| CoT seed | 94.6 | 81.4 | 72.8 | 83.0 | 85.2 | 58.9 | 84.6 | 60.4 | 77.62 | 1.0 |
| Structure only | 95.5 | 86.3 | 73.4 | 87.4 | 86.7 | 62.2 | 85.4 | 61.1 | 79.75 | 3.6 |
| Instruction only | 94.9 | 85.3 | 83.5 | 83.8 | 85.7 | 59.6 | 85.1 | 77.2 | 81.89 | 1.0 |
| Full | 95.8 | 88.4 | 83.5 | 88.4 | 87.2 | 62.8 | 85.9 | 78.5 | 83.82 | 3.4 |
- Full instruction-only is ; structure-only seed is . The library adds little on top: .
- This factorial removes the seed-selection confound, not the difference in calls. The instruction-only arm stays at 1.0 call and full search uses 3.4. It shows that a space that can add and organize calls beats one that can only reword a single call, not that learned structure beats ordinary sampling at the same number of calls; §5.3 says so.
- What the search builds: agreement-gated sampling (OlympiadBench, GPQA, SuperGPQA, MMLU-Pro), executable check-and-revise (MATH-500, LCB), and only a format instruction on TheoremQA.
6. Triggers: learned versus random placement (App. K.8, Table 19). We keep the learned sub-protocol and its firing rate, so mean calls are equal, and change only which problems it fires on.
| trigger | without | random | learned | learned random | inputs matched | |
|---|---|---|---|---|---|---|
| LCB (critique, 15%) | the seed's | 84.4 | 85.0 | 89.0 | no | |
| OlympiadBench (2 solves, 25%) | newly learned | 87.8 | 88.1 | 88.9 | yes | |
| SuperGPQA (3 solves, 84%) | seed's test, other side | 62.1 | 62.6 | 63.1 | yes |
The three rows support different conclusions:
- LCB: a random firing on an early-exit item gives the critique only the two agreeing programs, since the other candidates and verifier evidence exist only in the disagreement branch. Calls match but inputs do not, so this row does not isolate the trigger; point 2 is the LCB evidence.
- OlympiadBench: the search did produce a new condition, but its advantage over random placement is not significant.
- SuperGPQA: likewise; over the two benchmarks the average is .
- The reference column of Table 19 fires first on problems the program gets wrong, using test labels; we call it an error-first reference, not an oracle upper bound.
7. The re-search with unconditional patches, defined precisely (Table 21). The submitted description ("patch triggers forced to , seed conditions kept") was not self-consistent: a patch inserted inside a seed branch, like the LCB critique, still runs only when that branch is reached. We ran a restricted search space: every patch must have at a location that every run reaches, and seed programs keep their own conditions. This changes the local gate and the admissible locations together, so it is a re-search, not a counterfactual on . It gives 83.65 at 4.6 calls against 84.17 at 3.9; on LCB a critique outside the branch needs 8.0 calls, over budget, so the search settles for an unconditional executable check (87.0).
8. What the evidence supports, and what is not isolated.
| Claim | Status |
|---|---|
| TheoremQA and MuSR gains are instructions | supported; stated in the paper |
| MMLU-Pro gain is seed selection | supported; stated in the paper |
| a learned computation in an existing branch pays (LCB) | in this setting: vs. the seed; vs. a generic extra solve under the same trigger |
| full search beats matched instruction-only search | significant: with the library, from one CoT call, with more calls |
| ReProt beats instructed self-consistency with at least as many calls | on average only with LCB; LCB alone |
| newly learned triggers beat random ones at matched calls and inputs | not significant () |
| stability of the attribution arms across searches | not measured (one search per arm) |
Revised §5.3 and §6 state the last three rows as limits.
Summary of manuscript changes for W1b & Q2 (blue in the revised PDF):
| Location | Change |
|---|---|
| §5.3, Table 2a, App. K.1–K.2 | matched factorials; the call-count caveat on the CoT-seed factorial |
| §5.3, App. K.8, Table 18 | LCB with the seed's condition kept; the 9.7-point override is no longer evidence |
| Tables 19 and 21, App. K.8 | per-benchmark placement conclusions; "error-first" reference; unconditional-patch re-search redefined |
| Abstract, §1, §6 | the structure and trigger claims limited as above |
| §4.3, Eq. 7 | trigger set requires reaching ; costs are per-problem totals |
[W1a & Q3] The Contrast with AFlow's Conditional Logic, and the Role of the Free-Code Comparison [part 1/1]
Thank you for this very careful review. On Q3, you are right. Our direct answer: AFlow's workflows are code with conditional and test-dependent steps, as you note, and its search proceeds by single-step modifications (§5.2 and App. A.1 of Zhang et al.). We removed the claim that prior updates are unconditional, and we do not replace it with a contrast between small and whole-program edits. We now claim the distinction you identified: a constrained representation and a gated patch search. The free-code comparison is rerun with matched selection and a corrected implementation, and its evidential role is limited explicitly.
1. Revised text (blue in the PDF).
2. The distinction, stated without overreach.
We do not claim that AFlow cannot branch, that it changes many things at once, or that it cannot tell what an edit did. Nor is fix/break screening tied to our language: any two programs run on the same items yield paired fixes and breaks, and point 3 applies the screen to Python programs.
3. Q3, second half: the free-code comparison, matched (§5.3, App. K.7, Table 17). The rerun uses the final configuration and ReProt's selection rule (seeds and gate-accepted programs only, pooled search and validation accuracy), so no gate-rejected candidate can be selected. The
awaitbug is fixed, and both arms have a 600 s timeout. LCB is added as a branching benchmark.4. The evidential role, limited explicitly. App. K.7 now reads: "The comparison shows that, in our implementations, the typed language was cheaper to search and at least as accurate. It does not show that free code is less expressive …" We report this control apart from the patch-space factorials, because it alone changes the language.
5. What we no longer claim: that ReProt can represent programs prior methods cannot; that prior updates are unconditional or wholesale; that free code is less expressive.
Summary of manuscript changes for W1a & Q3 (blue in the revised PDF):