acceptodds
Under review as a conference paper at ICLR 2027

Learning What to Execute and When: Reasoning Protocols for LLM Inference

Abstract

Inference-time reasoning methods often extend a single language-model call through hand-designed procedures, such as voting, critique, program execution, or verification and revision. Learned multi-agent topologies optimize who communicates with whom, but have only matched hand-designed graphs under a common evaluation protocol. We introduce **REPROT**, which learns what to execute and when. REPROT represents a reasoning protocol as a stateful program over typed operations, specifying access to earlier results, operation composition, and runtime conditions that allocate extra computation to individual problems. Its language covers six common procedure families. A synthesizer examines search-set execution traces and proposes patches containing a diagnostic, a trigger condition, and a sub-protocol with defined visibility. A patch is accepted only if its net gain in problems fixed over problems broken passes a sign test on the full search set while meeting a mean-call budget. Protocol selection uses pooled search and disjoint validation data; the selected protocol is fixed before held-out testing. Across eight benchmarks, REPROT achieves the highest average accuracy, **84.2% at 3.9 calls per question**, compared with seventeen baselines reproduced under one protocol and the original ADAS and AFlow. The strongest hand-designed and learned-topology baselines score 79.7–80.3% at 5.2–11.0 calls; AFlow scores 82.6%. Paired comparisons with AFlow show four wins, four ties, and no losses. On TheoremQA and MuSR, REPROT scores 83.5% and 79.1%, versus 77.7% and 64.2% for the best reproduced baselines and 82.7% and 77.2% for AFlow. In both cases, the learned change is an instruction within a one- or three-call protocol. Free-code search finds a similar answer-format change on OlympiadBench. Condition overrides alter cost by up to 2.6×; forcing the first branch prevents LiveCodeBench escalation and lowers accuracy by 9.7 points.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Reasoning Protocol, Multi-Agent Reasoning, Agentic Workflow Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Wc45OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces REPROT, which searches over typed reasoning protocols specifying operations, visibility of intermediate artifacts, control flow, and answer commitment. A synthesizer proposes patches from labeled search-set execution digests, including state-conditioned computations and bounded instruction changes. Patches are screened using paired fix/break counts, a one-sided sign-test threshold, and a mean-call budget. Final selection uses pooled search and validation performance before freezing the protocol for testing. Experiments use one executor backbone on eight benchmarks, comparing seventeen reproduced baselines and the original ADAS and AFlow implementations. REPROT reports the highest average accuracy, 84.18%, at 3.9 calls per question.

Critical strengths
  1. Broad evaluation with meaningful aggregate improvements. The shared-backbone comparison covers eight tasks and includes direct runs of ADAS and AFlow, rather than relying only on topology baselines. REPROT averages 84.18% versus AFlow's 82.56%, and offers a favorable accuracy/call trade-off relative to the reproduced topology methods. The appendix also reports token usage and offline search cost. This makes the method practically interesting, while recognizing that AFlow remains cheaper online at 3.2 calls versus 3.9.
  2. An auditable representation and unusually explicit accounting of learned changes. Typed artifacts, enforced visibility, static validation, and commitment rules provide a clear implementation framework. The appendix records accepted patches, their fix/break counts, instructions, selection outcomes, and both successful and unsuccessful cases. The authors explicitly identify format-related gains on TheoremQA, instruction-only learning on MuSR, and selection of an unchanged seed on MMLU-Pro. The story-disjoint MuSR split is also an appropriate safeguard.
Critical weaknesses
  1. The distinctive contribution beyond existing workflow and prompt optimization is not adequately isolated. The introduction characterizes prior ADAS/AFlow updates as unconditional, but AFlow explicitly represents conditional logic and provides test-dependent correction workflows (Figure 3 and Appendix A.5 of Zhang et al., ICLR 2025). The plausible distinction is the constrained representation and gated patch-search procedure, rather than the ability to represent conditional computation. Yet structure-only and free-code controls differ in pools, rounds, digests, and, for free code, final selection rules; Appendix I also documents a dead branch caused by an await issue. The largest gains come from instructions, and MMLU-Pro returns an unchanged seed. Furthermore, the 9.7-point LiveCodeBench first-branch loss disables the cascade seed's existing escalation, so it does not isolate learned trigger improvement. These results demonstrate a useful overall system but do not yet establish the specific benefit of learning protocol structure over matched prompt optimization and seed selection.
  2. Search-time sampling semantics and the unit of statistical analysis need clarification. Appendix C.6 describes caching by model, prompt, and temperature, while Appendix D specifies identical prompts and temperatures across self-consistency branches. The cascade's initial solvers also both appear to use temperature 0 under the documented defaults. If identical requests share a cached response within a run, independent sampling and disagreement-triggered search behavior would be compromised. The actual implementation may preserve separate sample slots, but this is not explained and matters for the reported branch-dependent patches. Separately, sign tests pool three seed/item outcomes for each question, without retaining question-level dependence or story-level dependence in MuSR. The repeated evaluations of frozen REPROT protocols also do not measure search variability. These concerns warrant clarification and clustered reanalysis; they do not establish a test-cache bug or test-set leakage.
Questions for authors
  1. Does the cache key or lookup procedure include a node, sample slot, invocation index, or another distinction beyond the stated model/prompt/temperature tuple? Please explain how identical SC branches and the two initial cascade solvers retain independent samples within a run while preserving unchanged-prefix pairing across patches. A minimal search-time trace or implementation excerpt would be particularly useful.
  2. On a representative branching task and an instruction-dominated task, can you compare instruction-only optimization of a fixed seed with full search under the final pools, digests, rounds, synthesizer, gate, and selection rule? For LiveCodeBench, can you preserve the seed's cascade conditions and remove only the learned modification, reporting paired fixes/breaks and cost?
  3. How should the claimed contrast with unconditional prior workflows be revised given AFlow's conditional code examples? Can the free-code comparison use matched selection rules and a corrected implementation, or can its evidential role be explicitly limited?
  4. Do the AFlow advantage and per-benchmark conclusions persist under question-clustered resampling, with story-level clustering for MuSR? Please distinguish search seeds from execution seeds and clarify that the repeatedly reused p≤0.2 search gate is a heuristic rather than a calibrated discovery or generalization guarantee.
8 comments
AuthorsOpenReview ↗

[W1a & Q3] The Contrast with AFlow's Conditional Logic, and the Role of the Free-Code Comparison [part 1/1]

Thank you for this very careful review. On Q3, you are right. Our direct answer: AFlow's workflows are code with conditional and test-dependent steps, as you note, and its search proceeds by single-step modifications (§5.2 and App. A.1 of Zhang et al.). We removed the claim that prior updates are unconditional, and we do not replace it with a contrast between small and whole-program edits. We now claim the distinction you identified: a constrained representation and a gated patch search. The free-code comparison is rerun with matched selection and a corrected implementation, and its evidential role is limited explicitly.

1. Revised text (blue in the PDF).

  • §1, ¶3 (submitted): "Their updates are unconditional: a rewritten program or an added node runs on every input …"
  • §1, ¶3 (revised): "Code can express conditionals, and AFlow already learns conditional workflows through single-step modifications scored on validation data. Our contribution is an explicit patch interface that couples a runtime condition, declared visibility and a sub-protocol with paired fix/break screening under a mean-call budget; the screening also applies to code-level search, which a matched free-code control examines (Section 5.3)."
  • App. A (added): "The contrast drawn in this paper is therefore neither the ability to branch nor incremental editing. It is a specific interface and search design …"

2. The distinction, stated without overreach.

AFlowReProt
RepresentationPython workflow over operatorstyped protocol: nine operations, declared reads, conditions on runtime state only
Edit per roundone modification, written as codeone patch with explicit location, diagnostic, condition, sub-protocol and reads
Validityfound when the workflow runschecked statically before execution
How an edit is keptvalidation score of the modified workflow; success vs. its parent recordedpaired fixes vs. breaks on the search set and mean calls ≤B\leq B; selection on pooled search and validation data

We do not claim that AFlow cannot branch, that it changes many things at once, or that it cannot tell what an edit did. Nor is fix/break screening tied to our language: any two programs run on the same items yield paired fixes and breaks, and point 3 applies the screen to Python programs.

3. Q3, second half: the free-code comparison, matched (§5.3, App. K.7, Table 17). The rerun uses the final configuration and ReProt's selection rule (seeds and gate-accepted programs only, pooled search and validation accuracy), so no gate-rejected candidate can be selected. The await bug is fixed, and both arms have a 600 s timeout. LCB is added as a branching benchmark.

3 seeds; accuracy @ callsOlympiadBenchMMLU-ProLCB
Free code (matched)88.3 ± 0.6 @ 2.986.0 ± 0.5 @ 6.086.3 ± 1.7 @ 3.4
ReProt88.9 ± 0.6 @ 3.586.2 ± 0.4 @ 6.089.0 ± 2.0 @ 2.9
Free code: failed candidates9 / 506 / 5016 / 50
Uncached executor calls, free code vs. ReProt22.9k vs. 15.6k88.1k vs. 67.6k9.6k vs. 5.7k
  • Accepted rewrites. OlympiadBench: an answer-format instruction and one extra solve when the vote is not unanimous. MMLU-Pro: none, so the Debate seed is selected, as in the typed search. LCB: a public-test check that revises up to twice (conditional, but not the disagreement critique).
  • Failures, defined. 31 of 150 free-code candidates had a workflow-level failure: 14 uncaught exceptions, 9 timeouts and 8 branches found unreachable after execution. None of the 150 typed candidates did; 9 were rejected statically. API errors are retried in both arms and not counted. We do not claim that typed programs can never fail.
  • Cost. Our free-code implementation caches whole requests and obtained less reuse; another could reuse more, so the cost gap is not attributable to the language alone.

4. The evidential role, limited explicitly. App. K.7 now reads: "The comparison shows that, in our implementations, the typed language was cheaper to search and at least as accurate. It does not show that free code is less expressive …" We report this control apart from the patch-space factorials, because it alone changes the language.

5. What we no longer claim: that ReProt can represent programs prior methods cannot; that prior updates are unconditional or wholesale; that free code is less expressive.

Summary of manuscript changes for W1a & Q3 (blue in the revised PDF):

LocationChange
§1 ¶3, §2, App. Acontrast restated as above; "unconditional" removed
§5.3matched free-code result (within seed variation on two benchmarks, 86.3 vs. 89.0 on LCB) replaces the † row
App. I, K.7submitted free-code control superseded by the matched rerun (Table 17), with the limits above
AuthorsOpenReview ↗

[W1b & Q2] Isolating Learned Structure from Instructions and Seed Selection [part 1/2]

Thank you for these controls. Our answer to Q2: we ran instruction-only optimization under the final pools, digests, rounds, synthesizer, gate and selection rule on all eight benchmarks. On the instruction-dominated tasks (TheoremQA, MuSR), full search performs similarly to instruction-only search (83.5 vs. 83.2; 79.1 vs. 78.6), and the paper now calls these instruction gains. On LiveCodeBench, keeping the seed's cascade condition and removing only the learned modification costs 4.6 points (34 fixes vs. 10 breaks) for 0.1 mean calls. Beyond LiveCodeBench the structural contribution is positive but smaller, and two parts of the mechanism are not isolated; part 2/2 lists them.

1. Instruction-only optimization under the final setup (§5.3, Table 2a, App. K.1, Table 12; three seeds):

MATHOlympThQALCBGPQASGPQAMMLU-PMuSRAvgCalls
Seed selection only95.786.072.986.586.462.186.260.379.504.5
Structure only95.686.673.586.386.762.586.261.479.864.3
Instruction only95.787.883.286.186.462.286.278.683.263.4
Full95.988.983.589.087.763.186.279.184.173.9
  • Instruction-dominated tasks (TheoremQA, MuSR): §5.3 now says: "Instruction-only search is within 0.5 points of the full search on TheoremQA and MuSR, whose learned changes are single instructions (answer format; story tracking)."
  • Overall: full − instruction-only is +0.90 [0.13, 1.69], and +1.3 [0.2, 2.3] on the five benchmarks whose selected programs contain structural patches.
  • LCB alone: +2.9 [−0.6, 6.3], not significant against this arm; point 2 gives the cleaner LCB comparison.
  • MMLU-Pro: all four arms return the unpatched Debate seed. Its gain over the PHP root is a selection effect, not a structural one.
  • The instruction-only arm is not structure-free: it inherits hand-designed structure through the seeds it selects (Eq. 19); part 2/2 removes the library.

2. Q2, LiveCodeBench with the seed's cascade condition kept (§5.3, App. K.8, Table 18). You rightly note that the 9.7-point first-branch loss also disables the cascade's own escalation. We now keep the seed and its agree(c1, c2) test and vary only the learned critique:

LCB (3 seeds)AccCalls
P0P_0: cascade seed, learned patch removed84.4 ± 1.22.7
P∗P^{\ast}: critique in the disagreement branch89.0 ± 2.02.9
same trigger, one generic extra solve and a six-way vote85.72.9
critique on every item (with d1–d3, v1, v2 run everywhere)87.8 ± 2.08.0
P∗P^{\ast}, c2 at τ=0.7\tau=0.788.6 ± 1.73.3
  • Paired fixes and breaks: P∗P^{\ast} versus P0P_0 gives 34 fixes and 10 breaks pooled over three seeds; the question-clustered difference is +4.6 [1.4, 7.9].
  • Cost: +0.15 mean calls: the critique runs on the 77 of 525 outcomes where the first two programs disagree.
  • What LCB learned, and what it did not. The search learned the critique, its reads (programs and verifier evidence, not full reasoning) and its place in the seed's existing disagreement branch. It did not learn the condition; that is the seed's.
  • Under the same cascade trigger and mean calls, replacing the learned critique with a generic extra solve reduces accuracy by 3.2 points (95% question-clustered CI [0.2, 6.3]; 467 vs. 450 of 525). This supports the learned sub-protocol in this setting; it does not establish a general advantage over self-consistency or newly learned trigger predicates. Calls are matched, tokens are not.
  • Run everywhere, the critique costs 2.8× the calls; on items where the first two programs agree it fixes 11 and breaks 17.
  • The submitted override stays in Table 11 with your caveat (App. H.2); it is no longer our evidence.

3. Difficulty (Table 20). The critique fires on 2%, 10% and 36% of easy, medium and hard LCB items.

4. The same instructions with at least as many calls (§5.3, Table 2b, Table 14). Self-consistency receives ReProt's learned instructions and k=⌈ReProt’s mean calls⌉k=\lceil\text{ReProt's mean calls}\rceil samples per benchmark:

+ ReProt's instructionsAvgCallsReProt − method
Self-consistency, k≥ReProt’s callsk\geq\text{ReProt's calls}83.354.3+0.82 [0.08, 1.57]

On LCB the learned instruction belongs to the critique node, which self-consistency lacks, so this control is not fully matched; without LCB the difference is +0.42 [−0.05, 0.90]. On TheoremQA (k=1k=1) and MuSR, the self-consistency program is P∗P^{\ast} itself. We read the control as a bound: with the same instructions and at least as many calls, plain sampling comes within 0.82 points of ReProt on average, a difference significant only with LCB included. On LCB alone it is +3.6 [−1.3, 8.6], not significant; the call-matched LCB evidence is the generic-solve row of point 2.

AuthorsOpenReview ↗

[W1b & Q2] Isolating Learned Structure from Instructions and Seed Selection [part 2/2]

5. Removing the hand-designed library (§5.3, Table 2a, App. K.2, Table 13). Root and archive contain only s1 = solve[cot](problem); commit s1.

From one CoT callMATHOlympThQALCBGPQASGPQAMMLU-PMuSRAvgCalls
CoT seed94.681.472.883.085.258.984.660.477.621.0
Structure only95.586.373.487.486.762.285.461.179.753.6
Instruction only94.985.383.583.885.759.685.177.281.891.0
Full95.888.483.588.487.262.885.978.583.823.4
  • Full −- instruction-only is +1.94[1.12,2.69]+1.94[1.12,2.69]; structure-only −- seed is +2.13[1.36,2.97]+2.13[1.36,2.97]. The library adds little on top: +0.34[−0.45,1.16]+0.34[-0.45,1.16].
  • This factorial removes the seed-selection confound, not the difference in calls. The instruction-only arm stays at 1.0 call and full search uses 3.4. It shows that a space that can add and organize calls beats one that can only reword a single call, not that learned structure beats ordinary sampling at the same number of calls; §5.3 says so.
  • What the search builds: agreement-gated sampling (OlympiadBench, GPQA, SuperGPQA, MMLU-Pro), executable check-and-revise (MATH-500, LCB), and only a format instruction on TheoremQA.

6. Triggers: learned versus random placement (App. K.8, Table 19). We keep the learned sub-protocol QQ and its firing rate, so mean calls are equal, and change only which problems it fires on.

triggerwithout QQrandomlearnedlearned −- randominputs matched
LCB (critique, 15%)the seed's84.485.089.0+4.0[0.8,7.0]+4.0[0.8,7.0]no
OlympiadBench (2 solves, 25%)newly learned87.888.188.9+0.8[−0.4,2.1]+0.8[-0.4,2.1]yes
SuperGPQA (3 solves, 84%)seed's test, other side62.162.663.1+0.5[−0.2,1.3]+0.5[-0.2,1.3]yes

The three rows support different conclusions:

  • LCB: a random firing on an early-exit item gives the critique only the two agreeing programs, since the other candidates and verifier evidence exist only in the disagreement branch. Calls match but inputs do not, so this row does not isolate the trigger; point 2 is the LCB evidence.
  • OlympiadBench: the search did produce a new condition, but its advantage over random placement is not significant.
  • SuperGPQA: likewise; over the two benchmarks the average is +0.65[−0.07,1.39]+0.65[-0.07,1.39].
  • The reference column of Table 19 fires first on problems the program gets wrong, using test labels; we call it an error-first reference, not an oracle upper bound.

7. The re-search with unconditional patches, defined precisely (Table 21). The submitted description ("patch triggers forced to ⊤\top, seed conditions kept") was not self-consistent: a patch inserted inside a seed branch, like the LCB critique, still runs only when that branch is reached. We ran a restricted search space: every patch must have g≡⊤g\equiv\top at a location that every run reaches, and seed programs keep their own conditions. This changes the local gate and the admissible locations together, so it is a re-search, not a counterfactual on P∗P^{\ast}. It gives 83.65 at 4.6 calls against 84.17 at 3.9; on LCB a critique outside the branch needs 8.0 calls, over budget, so the search settles for an unconditional executable check (87.0).

8. What the evidence supports, and what is not isolated.

ClaimStatus
TheoremQA and MuSR gains are instructionssupported; stated in the paper
MMLU-Pro gain is seed selectionsupported; stated in the paper
a learned computation in an existing branch pays (LCB)in this setting: +4.6[1.4,7.9]+4.6[1.4,7.9] vs. the seed; +3.2[0.2,6.3]+3.2[0.2,6.3] vs. a generic extra solve under the same trigger
full search beats matched instruction-only searchsignificant: +0.90+0.90 with the library, +1.94+1.94 from one CoT call, with more calls
ReProt beats instructed self-consistency with at least as many callson average only with LCB; LCB alone +3.6[−1.3,8.6]+3.6[-1.3,8.6]
newly learned triggers beat random ones at matched calls and inputsnot significant (+0.65[−0.07,1.39]+0.65[-0.07,1.39])
stability of the attribution arms across searchesnot measured (one search per arm)

Revised §5.3 and §6 state the last three rows as limits.

Summary of manuscript changes for W1b & Q2 (blue in the revised PDF):

LocationChange
§5.3, Table 2a, App. K.1–K.2matched factorials; the call-count caveat on the CoT-seed factorial
§5.3, App. K.8, Table 18LCB with the seed's condition kept; the 9.7-point override is no longer evidence
Tables 19 and 21, App. K.8per-benchmark placement conclusions; "error-first" reference; unconditional-patch re-search redefined
Abstract, §1, §6the structure and trigger claims limited as above
§4.3, Eq. 7trigger set requires reaching ℓ\ell; costs are per-problem totals
Reviewer_kREXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2(+1)
Presentation
2(+1)
Soundness
2
Read review
Paper summary

The authors propose ReProt, which searches over typed inference programs that specify reasoning operations, information access, and conditional execution. With this instantiation, an LLM proposes patches from execution traces, and candidates are filtered using a paired sign-test gate and a call budget before final selection. The method reports the highest average accuracy across eight benchmarks.

Critical strengths

S1. The experimental coverage is substantial. It has various benchmarks and baselines, and the original ADAS and AFlow provide a broad evaluation of the complete system.

S2. The performance gains are promising across many benchmarks and baselines, supposed that the empirical results are fully reproducible.

S3. The technical implementations are presented in considerable details and well-written in a technical view.

Critical weaknesses

W1. The paper does describe a substantial system and a large set of experiments, yet does not clearly establish which idea is new and which mechanism explains the gains.

W2. I don't see the central claims or arguments. The introduction moves from the limited gains of current works without clearly establishing the missing capability or learning problem. It then introduces its proposed method's contributions in many aspects without making their relative importance clear. The issue is not simply dense writing; I am left uncertain about the specific new knowledge the experiments are intended to establish.

W3. The structure-only and free-code searches use earlier configurations with different rounds and data pools. Appendix I further states that the free-code control uses a different final selection rule, including gate-rejected candidates. This makes it difficult to attribute differences specifically to the protocol language or instruction slot.

Questions for authors

Q1. I wonder that what is the paper's main claim or argument beyond successful automated workflow optimization?

Q2. How much of the gain remains under a matched prompt-only control and an answer-normalization sensitivity analysis?

9 comments
AuthorsOpenReview ↗

[W1, W2 & Q1] The Main Claim, Which Idea Is New, and Which Mechanism Explains the Gains [part 1/1]

Thank you for naming the missing core. Our answer to Q1: the main claim is a criterion, not a ranking. Whether added inference computation helps a problem depends on how likely its current answer is to be wrong where the computation runs, and on how often that computation repairs a wrong answer relative to how often it breaks a correct one; under non-interference (outcomes change only where a patch fires) and π(r+b)>0\pi(r+b) > 0, with firing share π\pi and repair and break rates rr and bb, a patch raises average accuracy if and only if the error rate ϵg\epsilon_g where it runs exceeds its break share b/(r+b)b/(r+b) (Proposition 1, §4.3). ReProt learns procedures as state-conditioned edits screened by their paired fixes and breaks under a call budget.

1. The missing capability and the learning problem (W2), as revised §1 states them:

  • Learned topologies vary who talks to whom; under one protocol they only match hand-designed graphs (80.3 and 80.1 vs. 80.2 for a complete graph). Paired outcomes show why: three learned topologies change 10.9–11.7% of a complete graph's outcomes, but fixes and breaks nearly cancel (Table 33).
  • Prompt optimizers change what a step says, not which steps run or when.
  • Workflow search (ADAS, AFlow) changes the procedure, and AFlow already learns conditional workflows through single-step edits. It leaves implicit when a step runs, what it may read and what an edit costs.
  • The learning problem is accuracy under a mean-call budget (Eq. 8), with edits whose condition, reads and sub-protocol are explicit and whose paired fixes, breaks and calls are measured on the search set (Eq. 9).

2. Which idea is new (W1). Not branching and not incremental editing: AFlow has both, and the submitted sentence that called prior updates "unconditional" is removed. New are (i) the decomposition Δ=π(r+b)(ϵg−β)\Delta=\pi(r+b)(\epsilon_g-\beta), which separates a property of the trigger (ϵg\epsilon_g) from a property of what is executed (rr, bb, β\beta), with the budget-optimal firing rule that ranks states by expected net gain per added call (Proposition 2, App. N); and (ii) a patch interface that couples a runtime condition, declared visibility and a sub-protocol, screened by paired fixes and breaks under a mean-call budget.

3. Which mechanism explains the gains (W1, Q1) (§5.3, Table 2, Table 33, Apps. K–N; question-clustered 95% intervals).

QuestionComparisonResultStatus
Is the gain answer format?all 25 methods rescored lenientlyReProt 84.19, AFlow 82.78TheoremQA lead disappears; average lead stays
Is it only prompts?full −- instruction-only search (library)+0.90[0.13,1.69]+0.90[0.13,1.69]; 3.9 vs. 3.4 callssignificant; more calls
same, from one CoT call+1.94[1.12,2.69]+1.94[1.12,2.69]; 3.4 vs. 1.0 callssignificant; more calls
Why do unconditional instructions raise average accuracy?paired fixes and breaks (Table 33)β=0.031,0.068\beta=0.031,0.068 vs. ϵ=0.272,0.396\epsilon=0.272,0.396mechanism analysis
What was learned on LCB?critique vs. seed without it (condition kept)+4.6[1.4,7.9]+4.6[1.4,7.9]; 34 fixes, 10 breaks in the branch; 11, 17 on agreeing itemssignificant
critique vs. a generic extra solve, same trigger+3.2[0.2,6.3]+3.2[0.2,6.3] (467 vs. 450 of 525)significant here; tokens not matched
Do learned triggers beat random ones?OlympiadBench / SuperGPQA, calls and inputs matched+0.8[−0.4,2.1]+0.8[-0.4,2.1] / +0.5[−0.2,1.3]+0.5[-0.2,1.3]not significant
  • Instructions are the largest single component (79.86 without the instruction slot) and explain TheoremQA and MuSR; they rarely break a correct answer (15 and 25 breaks), so applied unconditionally they raise average accuracy.
  • The LCB rows support the learned sub-protocol in this setting. They do not establish a general advantage over self-consistency, and learned triggers do not beat random ones at matched calls and inputs; with rr and bb fixed at random-placement values, even firing on wrong answers first would add just 2.4 and 2.7 points more (reference values; App. N.4).
  • One computation path: averages and differences use unrounded scores, rounded last; ReProt's average is 84.17, not the submitted 84.18 (App. L.4).

4. Reorganization (W2). §1 now states the gap, the central claim as a conditional criterion, the corrected relation to ADAS and AFlow, and three contributions; the last names four mechanisms, not point differences. Proposition 1 is stated in §4.3 with its conditions; §5.3 ends with the paired-outcome reanalysis (Table 33).

Summary of manuscript changes (blue in the revised PDF):

LocationChange
Abstract, §1, §6central claim as a criterion, with its conditions; corrected relation to AFlow; contributions name mechanisms, not point differences
§4.3, App. N, Table 33Propositions 1–2, corollaries, proofs, paired-outcome reanalysis
§5.2–§5.4reorganized around the attribution controls (Table 2)
AuthorsOpenReview ↗

[W3] Structure-Only and Free-Code Controls Rerun Under the Final Configuration and Selection Rule [part 1/1]

Thank you; you are right that the submitted structure-only and free-code controls used earlier configurations and the free-code control a different selection rule. Our answer: we reran both under the final configuration (R=10,K=5R=10,K=5, final pools and digests, the same synthesizer, gate and selection rule). The free-code control now uses ReProt's selection rule, so only seeds and gate-accepted programs are eligible, and its await bug is fixed. Every † row of the submission is superseded. The reruns isolate the instruction slot; for the language they support a narrower conclusion (point 3).

1. Matched factorial with the seed library (§5.3, Table 2a, App. K.1, Table 12). Only the kinds of patch the synthesizer may propose vary; three execution seeds.

ArmMATHOlympThQALCBGPQASGPQAMMLU-PMuSRAvgCalls
Seed selection only95.786.072.986.586.462.186.260.379.504.5
Structure-only95.686.673.586.386.762.586.261.479.864.3
Instruction-only95.787.883.286.186.462.286.278.683.263.4
Full (ReProt)95.988.983.589.087.763.186.279.184.173.9
  • The instruction slot is worth 4.31 points (84.17 vs. 79.86), concentrated on TheoremQA and MuSR. The submitted number (earlier configuration) was 79.03.
  • Full search scores +0.90 [0.13, 1.69] above instruction-only search, and +1.3 [0.2, 2.3] on the five benchmarks whose selected programs contain structural patches. On LCB alone the difference (+2.9 [−0.6, 6.3]) is not significant.
  • MMLU-Pro returns the unpatched Debate seed in all four arms; its gain over the PHP root is a selection effect.
  • The instruction-only arm is not structure-free: it inherits hand-designed structure through the seeds it selects. The single-CoT-seed factorial removes that (Table 13): 77.62 (seed), 79.75 (structure-only), 81.89 (instruction-only) and 83.82 (full), at 1.0, 3.6, 1.0 and 3.4 calls. The arms differ in calls, so this shows what a search that may add calls gains over one that may only reword one call.

2. Free-code control, matched (§5.3, App. K.7, Table 17). It uses the final configuration, ReProt's selection rule (seeds and gate-accepted programs only, pooled search and validation accuracy, ties broken by cost), the corrected implementation and a 600 s per-problem timeout in both arms. LCB is added for branching.

3 seeds; accuracy @ callsOlympiadBenchMMLU-ProLCB
Free code (matched)88.3 ± 0.6 @ 2.986.0 ± 0.5 @ 6.086.3 ± 1.7 @ 3.4
ReProt88.9 ± 0.6 @ 3.586.2 ± 0.4 @ 6.089.0 ± 2.0 @ 2.9
Free code: failed candidates9 / 506 / 5016 / 50
Uncached executor calls, free code vs. ReProt22.9k vs. 15.6k88.1k vs. 67.6k9.6k vs. 5.7k
  • Accepted rewrites. OlympiadBench: an answer-format instruction and one extra solve on a split vote. MMLU-Pro: none, so the Debate seed was selected. LCB: a public-test check with up to two revisions.
  • Failures, defined. 31 of 150 free-code candidates had a workflow-level failure: 14 uncaught exceptions, 9 timeouts and 8 branches found unreachable after execution. None of the 150 typed candidates did, and 9 were rejected statically.
  • Same rules in both arms: the free-code arm applies the same fix/break screen and Eq. 19 selection to whole programs, so no arm can select a gate-rejected candidate.
  • Corrected: ReProt's LCB s.d. is 2.0, not 2.3; its three-seed mean calls are 2.88 (the 2.8 of Table 11 is at seed 42); see App. L.4.

3. What the reruns do and do not support.

  • Instruction slot: isolated. The submission said its comparison "bounds but does not isolate" the slot's effect; the matched factorial replaces that statement.
  • Language: a narrower claim. Free code is within seed variation on OlympiadBench and MMLU-Pro and trails on LCB (three-benchmark average 86.87 vs. 88.02). It can express every ReProt protocol. The control shows that, in our implementations, the typed language was cheaper to search and at least as accurate, not that free code is less expressive.
  • Cost: not attributable to the language alone. Our free-code implementation obtained less cache reuse; another implementation could reuse more (App. K.7).

Summary of manuscript changes for W3 (blue in the revised PDF):

LocationChange
Old Table 2 (now Table 11)kept for reference; App. H.2 states that the † controls are superseded
§5.3, Table 2amatched factorials replace "bounds but does not isolate the slot's effect"
App. Ifree-code control marked as superseded by App. K.7
App. K.1, K.2, K.7per-seed results (Tables 12, 13, 17); limits of the free-code comparison stated
AuthorsOpenReview ↗

[Q2] Matched Prompt-Only Control and Answer-Normalization Sensitivity [part 1/2]

Thank you for these two questions. Our answer: under a matched prompt-only control (the same pipeline restricted to instruction patches), full search still scores +0.90 [0.13, 1.69] higher with the seed library and +1.94 [1.12, 2.69] higher from a single CoT call, the latter with 3.4 instead of 1.0 calls. Under lenient answer normalization applied to all 25 methods, ReProt's TheoremQA lead disappears and its average lead over AFlow shrinks from 1.61 to 1.41 points (84.19 vs. 82.78). What remains sits on the benchmarks whose learned programs add conditional computation.

1. Matched prompt-only control (§5.3, Table 2a, App. K.1–K.2, Tables 12–13). Instruction-only search keeps pools, digests, RR, KK, synthesizer, gate and selection identical and allows only set_instruction patches. Instruction-only → full:

MATHOlympThQALCBGPQASGPQAMMLU-PMuSRAvgCalls
with seed library95.7→95.987.8→88.983.2→83.586.1→89.086.4→87.762.2→63.186.2→86.278.6→79.183.26→84.173.4→3.9
from one CoT call94.9→95.885.3→88.483.5→83.583.8→88.485.7→87.259.6→62.885.1→85.977.2→78.581.89→83.821.0→3.4
  • TheoremQA and MuSR are instruction gains: the prompt-only arm is within 0.5 points of full search there (83.2 vs. 83.5; 78.6 vs. 79.1), and the paper labels them so.
  • The remaining gain concentrates on LCB, OlympiadBench, GPQA and SuperGPQA in both settings.
  • With the library, the prompt-only arm is not structure-free. It selects hand-designed seeds: Debate on MATH-500 and MMLU-Pro, SC-3 on OlympiadBench and MuSR, the cascade on LCB and SuperGPQA, PHP on GPQA. On MMLU-Pro both arms return the same unpatched Debate seed.
  • From one CoT call, nothing is inherited, but the arms differ in calls. The +1.94 shows what a search that may add and organize calls gains over one that may only reword a single call; it is not a call-matched comparison.

2. How much of the gain remains under prompt-only control:

ComparisonFull − prompt-onlyCalls
with the seed library+0.90 [0.13, 1.69]3.9 vs. 3.4
… on the five benchmarks with structural patches+1.3 [0.2, 2.3]
… on LCB alone+2.9 [−0.6, 6.3]
from one CoT call+1.94 [1.12, 2.69]3.4 vs. 1.0

3. External prompt optimizers on the same root (Table 1, App. K.5). GEPA optimizes the prompt of every LLM-calling node of each benchmark's root P0P_0, keeping its structure; MIPROv2 optimizes instructions and up to four demonstrations per node. Both get rollout budgets equal to our search's executor calls (±5%).

MATHOlympThQALCBGPQASGPQAMMLU-PMuSRAvgCalls
MIPROv2 on P0P_095.387.081.684.685.561.885.575.982.142.4
GEPA on P0P_095.487.482.485.185.962.085.676.582.542.4
ReProt95.988.983.589.087.763.186.279.184.173.9

ReProt leads GEPA by +1.63 [0.95, 2.36] and MIPROv2 by +2.03 [1.29, 2.78] on the average, at 1.5 more calls. Per benchmark, only LCB is a significant win over either (1–7–0; Figure 4). Both find the same kinds of instruction as ReProt's (answer format, story tracking).

4. A prompt-matched control with at least as many calls (§5.3, Table 2b, Table 14). Baselines receive ReProt's learned instructions on their answer-producing calls. Self-consistency samples k=⌈ReProt’s mean calls⌉k=\lceil\text{ReProt's mean calls}\rceil (5, 4, 1, 3, 6, 6, 6, 3), so it never has the smaller budget.

+ ReProt's instructionsMATHOlympThQALCBGPQASGPQAMMLU-PMuSRAvgCalls
Self-consistency96.188.083.585.386.762.585.679.183.354.3
AFlow95.987.683.488.285.061.684.878.083.053.2
ReProt95.988.983.589.087.763.186.279.184.173.9
  • ReProt leads by +0.82 [0.08, 1.57] and +1.11 [0.33, 1.93].
  • This control is not fully matched. On LCB the learned instruction belongs to the critique node, which self-consistency lacks. Without LCB, ReProt − self-consistency is +0.42 [−0.05, 0.90], not significant. On TheoremQA and MuSR the self-consistency program is ReProt's own program.
  • Our reading: given the same instructions and at least as many calls, the average advantage over plain sampling is significant only with LCB included, and on LCB alone ReProt − self-consistency is +3.6 [−1.3, 8.6], also not significant. The call-matched LCB evidence is narrower: replacing the learned critique with a generic extra solve under the same cascade trigger reduces accuracy by 3.2 points ([0.2, 6.3]; App. K.8), which supports the learned sub-protocol in that setting alone.
Reviewer_Qz1YOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ReProt to optimize LLM workflows, including which operations to run, what information they can access, and when to use additional computation. The results are generally better than the baselines. However, it is unclear how much improvement comes from learning the workflow itself, since several large gains come from changing prompts or answer formats.

Critical strengths

Generally, the proposed method achieves good results across eight benchmarks. The authors also provide detailed execution traces and discuss failure cases, which help explain where the improvements come from.

Critical weaknesses

I have the following concerns.

  • I think the novelty is limited. Automatically optimizing LLM workflows has been studied in ADAS and AFlow. The explicit visibility rules and patch acceptance procedure are useful design choices, but I am not sure whether we nees such a complex framework.

  • The comparison with simpler prompt optimization is missing. On TheoremQA, 52 of 66 fixes only correct the answer format, while the original answers already have the right values. On MuSR, the final method is still three independent solutions followed by voting. These results do not clearly demonstrate the benefit of learning a whole execution structure.

  • The usefulness for stronger models remains unclear. All experiments use gpt-5.6-luna as the executor. Evaluating a fixed workflow with three random seeds does not show whether the search itself consistently finds good workflows, or whether the improvements transfer to other models.

  • The cost advantage over the strongest baseline is unclear. Compared with AFlow, the method improves average accuracy from 82.56 to 84.18, but increases calls per question from 3.2 to 3.9. The additional search cost should also be considered.

Questions for authors

Please refer to the above part.

7 comments
AuthorsOpenReview ↗

[W1] Novelty Relative to ADAS and AFlow, and Whether the Framework Is Overbuilt [part 1/1]

Thank you for pressing on novelty and necessity. Our direct answer: AFlow already learns conditional workflows through single-step modifications, so neither branching nor incremental editing is our novelty, and the revised paper no longer implies it. ReProt contributes a specific interface and search design: a patch whose runtime condition, declared reads and sub-protocol are explicit fields, screened by paired fixes and breaks under a mean-call budget.

1. Corrected positioning (§1, §2, App. A; blue in the PDF). The submitted sentence "Their updates are unconditional …" is removed. §1 now reads: "Code can express conditionals, and AFlow already learns conditional workflows through single-step modifications scored on validation data. Our contribution is an explicit patch interface that couples a runtime condition, declared visibility and a sub-protocol with paired fix/break screening under a mean-call budget …"

ADASAFlowReProt
Searched objectagent codeoperator workflow (code)typed protocol
Edit per rounda new agent programone modification (an operator or a prompt)one patch δ=(ℓ,d,g,Q,VQ)\delta=(\ell,d,g,Q,V_Q)
Conditional stepsexpressible in codeexpressible in codecondition gg is a field of the patch
What a step readsimplicit in codeimplicit in codedeclared per read, checked statically
How an edit is keptarchive, by fitnessvalidation score; success vs. parent recordedpaired fixes vs. breaks on the search set, mean calls ≤B\leq B

We do not claim that prior methods cannot make small edits, branch, or record what an edit did, nor that fix/break screening needs our language: the free-code control (point 3) applies the same gate to Python programs.

2. What an accepted patch looks like (LiveCodeBench, App. I, Table 8):

dis_synth = solve(reads=[st1.answers, st2.answers, v1.evidence, v2.evidence])
            {"style":"critique", "instruction": L}
commit dis_synth > agg1        # (n_fix, n_brk, p, ĉ) = (4, 0, .0625, 3.01)

The cascade seed already branches when its first two programs disagree (about 15% of items). The search learned the critique, its reads (answers and verifier evidence, not full reasoning) and its place in that branch; it did not learn the condition. With the seed's condition kept and only this patch removed, LCB falls from 89.0 to 84.4 (34 fixes vs. 10 breaks; +4.6 [1.4, 7.9]; Table 18).

3. Is each part needed? (§5.3; Tables 12, 16, 17, 21; three execution seeds)

ChangeAvgCallsEffect
ReProt84.173.9
no instruction slot79.864.3−4.31; the largest component (TheoremQA, MuSR)
instruction patches only83.263.4−0.90 [−1.69, −0.13]
any net gain accepted (α=1\alpha=1)83.354.3−0.82; 65 patches accepted, instead of 14
unconditional patches only83.654.6−0.5 at +0.7 calls
all reads open / all isolated83.59 / 83.923.9LCB −3.8 [−7.1, −0.8] / MMLU-Pro −2.1 [−3.5, −0.8]
no budget (B=∞B=\infty)84.255.8+0.08 for 1.5× the calls
free code, same gate and selection (3 benchmarks)86.87 vs. 88.02within seed noise on two; 86.3 vs. 89.0 on LCB
  • 84.17, not the submitted 84.18: all averages and differences are now computed from unrounded per-item scores and rounded last (App. L.4); the submission averaged rounded benchmark means.
  • Most rows re-run the search in a changed search space, so the selected programs differ; only the visibility rows edit the same program. "Unconditional patches only" permits only g≡⊤g\equiv\top at locations every run reaches, so it changes the gate and the admissible locations together (App. K.8).
  • The acceptance rule is a screening heuristic, not a guarantee: (3, 0) passes at p=0.125p=0.125 (§4.4). The 50 extra patches that α=1\alpha=1 admits show no detectable aggregate benefit on test (162 fixes vs. 158 breaks).
  • Visibility is site-specific. With full reasoning, the LCB critique sides with the first program on 78% of disagreements (49% with answer-and-evidence reads); isolated reads hurt Debate (MMLU-Pro).
  • Typed versus free code. 31 of 150 free-code candidates crashed, timed out or had unreachable branches; none of the 150 typed candidates did (9 were rejected statically; see App. K.7).

4. Is it complex to use? One configuration (R=10,K=5,α=0.2,B=6R=10,K=5,\alpha=0.2,B=6) runs all ten benchmarks; nothing is tuned per task.

Summary of manuscript changes for W1 (blue in the revised PDF):

LocationChange
§1 ¶3, §2, App. AAFlow described as conditional and single-step; contribution stated as the patch interface with fix/break screening
§4.4, App. K.6gate stated as a screening heuristic, with its limits
§5.3, App. K.8component controls; the unconditional-patch re-search defined precisely
App. K.7free-code control under the same gate; its evidential role limited
AuthorsOpenReview ↗

[W2] Comparison with Simpler Prompt Optimization, TheoremQA Format Fixes, and MuSR [part 2/2]

4. Search from a single CoT call (§5.3, Table 2a, Table 13). Root and archive contain only s1 = solve[cot](problem), so no hand-designed structure is inherited.

From one CoT callMATHOlympThQALCBGPQASGPQAMMLU-PMuSRAvgCalls
CoT seed94.681.472.883.085.258.984.660.477.621.0
Structure-only95.586.373.487.486.762.285.461.179.753.6
Instruction-only94.985.383.583.885.759.685.177.281.891.0
Full95.888.483.588.487.262.885.978.583.823.4
  • Full search scores +1.94[1.12,2.69]+1.94[1.12,2.69] above instruction-only search, on seven of the eight benchmarks; the tie is TheoremQA.
  • This is not a call-matched comparison: the arms use 3.4 and 1.0 calls. It shows that a space that can add and organize calls beats one that can only reword a single call, not that learned structure beats plain sampling at the same calls (see points 3 and 5). §5.3 states this limit.
  • What the search builds: agreement-gated sampling (OlympiadBench, GPQA, SuperGPQA, MMLU-Pro) and executable check-and-revise (MATH-500, LCB).

5. The clearest structural evidence: LiveCodeBench (§5.3, App. K.8, Tables 18–19). We keep the cascade seed and its disagreement condition and remove only the learned critique:

LCB (3 seeds)AccCalls
cascade seed, learned critique removed84.4±1.284.4\pm1.22.7
ReProt: critique in the disagreement branch89.0±2.089.0\pm2.02.9
same trigger, one generic extra solve instead of the critique85.72.9
critique on every item (five programs, two verifications)87.8±2.087.8\pm2.08.0

Against the seed without it, the critique fixes 34 items and breaks 10 over the three seeds (+4.6[1.4,7.9]+4.6[1.4,7.9]) for 0.15 more calls. Replacing the learned critique with a generic extra solve under the same cascade trigger reduces accuracy by 3.2 points (95% question-clustered CI [0.2,6.3][0.2,6.3]; 467 vs. 450 of 525; calls matched, tokens not). This supports the learned sub-protocol in this setting; it does not establish a general advantage over self-consistency or newly learned trigger predicates. (S.d. 2.0 corrects the submitted 2.3; calls are three-seed means (2.88), where 2.8 was at seed 42; App. L.4.)

6. Every fix accounted for (Table 9, Table 27). Of ReProt's 238 fixes over its roots at seed 42:

fixesbreaksnet
form-only (right value, rejected form)56 (52 on TheoremQA)056
value-level18261121

Of the 182 value-level fixes, 43 select a candidate the root already had and 139 produce an answer absent from the root's trace. These counts are relative to ReProt's own roots and do not decompose its lead over other methods; the lenient rescoring in part 1/2 is our main answer on format.

7. MuSR is an instruction gain, and the paper labels it so. The selected program is SC-3 with one learned instruction ("track the story mechanically, per subtask").

  • No story is shared between the search, validation and test partitions.
  • Most fixes are not format changes: 85 of the 106 MuSR fixes are generation failures: no root candidate chose the right option.
  • Against learned baselines: ReProt scores 79.1, GEPA 76.5, MIPROv2 75.9 and MASS 74.3; under story clustering only the MASS difference is significant.

8. Two benchmarks where formats are fixed (added; Table 2c, Table 15): AIME 2025+2026 (60 problems, integer answers) and BigCodeBench-Hard (148 tasks, official tests). Accuracy (mean calls):

MethodAIME 2025+26BCB-Hard
AFlow69.4 (3.6)37.4 (3.1)
Instruction-only search67.8 (3.0)35.8 (2.8)
ReProt76.7 (5.4)41.4 (3.3)

Instructions add +0.6+0.6 and +0.7+0.7 over the roots, and full search adds +8.9[1.1,17.2]+8.9[1.1,17.2] and +5.6[1.3,10.2]+5.6[1.3,10.2] over instruction-only search, with more calls on AIME (5.4 vs. 3.0). The AIME program runs an executed check only when its three solves disagree.

To sum up. Instructions are the largest source of gain and explain TheoremQA and MuSR. Search that can also add conditional computation scores higher than instruction-only search in every setting tested, with more calls. The call-matched evidence is local to the LiveCodeBench critique (in point 5).

Summary of manuscript changes for W2 (blue in the revised PDF):

LocationChange
Abstract, §1, §6"structure drives the gain" removed; the statements above are now in §5.3 with calls
§5.1, Table 1GEPA, MIPROv2, MaAS, MASS and Adaptive-Consistency added
§5.2, App. L.2"Answer format" paragraph; lenient rescoring of all 25 methods (Table 28)
§5.3, Table 2factorials, the call-matched controls and their limits, fixed-format benchmarks
AuthorsOpenReview ↗

[W2] Comparison with Simpler Prompt Optimization, TheoremQA Format Fixes, and MuSR [part 2/2]

4. Search from a single CoT call (§5.3, Table 2a, Table 13). Root and archive contain only s1 = solve[cot](problem), so no hand-designed structure is inherited.

From one CoT callMATHOlympThQALCBGPQASGPQAMMLU-PMuSRAvgCalls
CoT seed94.681.472.883.085.258.984.660.477.621.0
Structure-only95.586.373.487.486.762.285.461.179.753.6
Instruction-only94.985.383.583.885.759.685.177.281.891.0
Full95.888.483.588.487.262.885.978.583.823.4
  • Full search scores +1.94 [1.12, 2.69] above instruction-only search, on seven of the eight benchmarks; the tie is TheoremQA.
  • This is not a call-matched comparison: the arms use 3.4 and 1.0 calls. It shows that a space that can add and organize calls beats one that can only reword a single call, not that learned structure beats plain sampling at the same calls (see points 3 and 5). §5.3 states this limit.
  • What the search builds: agreement-gated sampling (OlympiadBench, GPQA, SuperGPQA, MMLU-Pro) and executable check-and-revise (MATH-500, LCB).

5. The clearest structural evidence: LiveCodeBench (§5.3, App. K.8, Tables 18–19). We keep the cascade seed and its disagreement condition and remove only the learned critique:

LCB (3 seeds)AccCalls
cascade seed, learned critique removed84.4 ± 1.22.7
ReProt: critique in the disagreement branch89.0 ± 2.02.9
same trigger, one generic extra solve instead of the critique85.72.9
critique on every item (five programs, two verifications)87.8 ± 2.08.0

Against the seed without it, the critique fixes 34 items and breaks 10 over the three seeds (+4.6 [1.4, 7.9]) for 0.15 more calls. Replacing the learned critique with a generic extra solve under the same cascade trigger reduces accuracy by 3.2 points (95% question-clustered CI [0.2, 6.3]; 467 vs. 450 of 525; calls matched, tokens not). This supports the learned sub-protocol in this setting; it does not establish a general advantage over self-consistency or newly learned trigger predicates. (S.d. 2.0 corrects the submitted 2.3; calls are three-seed means (2.88), where 2.8 was at seed 42; App. L.4.)

6. Every fix accounted for (Table 9, Table 27). Of ReProt's 238 fixes over its roots at seed 42:

fixesbreaksnet
form-only (right value, rejected form)56 (52 on TheoremQA)056
value-level18261121

Of the 182 value-level fixes, 43 select a candidate the root already had and 139 produce an answer absent from the root's trace. These counts are relative to ReProt's own roots and do not decompose its lead over other methods; the lenient rescoring in part 1/2 is our main answer on format.

7. MuSR is an instruction gain, and the paper labels it so. The selected program is SC-3 with one learned instruction ("track the story mechanically, per subtask").

  • No story is shared between the search, validation and test partitions.
  • Most fixes are not format changes: 85 of the 106 MuSR fixes are generation failures: no root candidate chose the right option.
  • Against learned baselines: ReProt scores 79.1, GEPA 76.5, MIPROv2 75.9 and MASS 74.3; under story clustering only the MASS difference is significant.

8. Two benchmarks where formats are fixed (added; Table 2c, Table 15): AIME 2025+2026 (60 problems, integer answers) and BigCodeBench-Hard (148 tasks, official tests). Accuracy (mean calls):

MethodAIME 2025+26BCB-Hard
AFlow69.4 (3.6)37.4 (3.1)
Instruction-only search67.8 (3.0)35.8 (2.8)
ReProt76.7 (5.4)41.4 (3.3)

Instructions add +0.6 and +0.7 over the roots, and full search adds +8.9 [1.1, 17.2] and +5.6 [1.3, 10.2] over instruction-only search, with more calls on AIME (5.4 vs. 3.0). The AIME program runs an executed check only when its three solves disagree.

To sum up. Instructions are the largest source of gain and explain TheoremQA and MuSR. Search that can also add conditional computation scores higher than instruction-only search in every setting tested, with more calls. The call-matched evidence is local to the LiveCodeBench critique (in point 5).

Summary of manuscript changes for W2 (blue in the revised PDF):

LocationChange
Abstract, §1 (F3), §6"structure drives the gain" replaced by the statements above, with calls stated
§5.1, Table 1GEPA, MIPROv2, MaAS, MASS and Adaptive-Consistency added
§5.2, App. L.2"Answer format" paragraph; lenient rescoring of all 25 methods (Table 28)
§5.3, Table 2factorials, the call-matched controls and their limits, fixed-format benchmarks

Comments on the paper

2 comments
AuthorsOpenReview ↗

[General Response] What the Revision Adds, What It Concedes, and Where Each Concern Is Answered [part 1/2]

We thank all three reviewers for their detailed feedback. The revision explains when added inference computation improves accuracy, adds controls to distinguish instruction, structure, and seed-selection effects, and narrows the claims about learned triggers and compute cost. This part maps the evidence to the reviewers’ concerns. Part 2 summarizes what we now claim and the scope of the evidence.

New evidence and reanalysis, by concern

ConcernRaised byWhat we foundWhere
What is the paper’s central finding?kREXProposition 1 relates the value of a patch to both the error rate where it runs and the repair and break rates of what it executes. The same LiveCodeBench critique produces 34 fixes and 10 breaks where candidate answers disagree, versus 11 fixes and 17 breaks where they agree.§4.3, Table 33, App. N
Do answer format or prompt edits explain the gains?Qz1Y, kREX, Wc45Lenient rescoring across all 25 methods removes the TheoremQA lead. The eight-benchmark average remains 84.19 versus 82.78 for AFlow. Learned instructions account for the TheoremQA and MuSR gains.§5.2, Table 28
Does full search add value beyond instructions and seed selection?Qz1Y, kREX, Wc45Full search leads instruction-only search by 0.90 points with the same seed library and by 1.94 points from a single CoT seed. Both comparisons use more calls. Against instructed self-consistency with at least as many calls, the average lead is 0.82 points. Without LiveCodeBench, the interval for the remaining 0.42-point lead includes zero.§5.3, Table 2, Tables 12–14
What do we learn from holding the trigger fixed?kREX, Wc45On LiveCodeBench, replacing a generic extra solve with the learned critique under the same trigger improves accuracy by 3.2 points. Calls are matched, while token use may differ. With both calls and inputs matched, learned triggers on OlympiadBench and SuperGPQA show no statistically significant gain over random placement.Tables 18–19
How should AFlow and free code be compared with ReProt?Qz1Y, kREX, Wc45We corrected the description of AFlow’s conditional single-step edits and reran the free-code control using the same gate and selection rule. In the three-benchmark comparison, 31 of 150 free-code candidates had workflow-level failures, versus zero of 150 typed candidates.§2, App. K.7
What are the search and inference costs?Qz1YThe main configuration uses 3.9 calls per question versus AFlow’s 3.2. A separate search under AFlow’s mean-call budget scores 83.59 at 2.6 calls, versus 82.56 at 3.2 for AFlow. We report offline executor costs separately.§5.4, Table 25, App. K.11
Are the paired comparisons and gate decisions reliable?Wc45With question-level clustering and story-level clustering for MuSR, the average AFlow margin is +1.61[0.79,2.46]+1.61[0.79,2.46] points and the per-benchmark record is 3 wins and 5 ties. A cache-free replay reproduces 81 of 84 gate decisions.Figure 4, Table 26, Apps. L and M

The revision also distinguishes independent searches from repeated execution of a frozen program, and transfer to another executor from rerunning the search with that executor. Those results are reported separately in §5.4 and Apps. K.9–K.10.

AuthorsOpenReview ↗

[General Response] What the Revision Adds, What It Concedes, and Where Each Concern Is Answered [part 2/2]

The revised paper makes a more specific claim: an inference edit helps when it runs on problems whose current answers are sufficiently likely to be wrong, given how often the edit repairs wrong answers and breaks correct ones. ReProt uses paired execution records to screen such edits under a call budget.

1. Claims we narrowed

  • Prompt and structure attribution. Learned instructions explain the TheoremQA and MuSR gains. The strongest direct evidence for learned structure is the LiveCodeBench comparison between the learned critique and a generic extra solve under the same trigger. A general advantage over instructed self-consistency at comparable call counts remains unestablished.
  • Learned triggers and communication topology. The matched-input trigger comparisons on OlympiadBench and SuperGPQA are inconclusive. In the topology comparison, three learned topologies change 10.9–11.7% of the complete graph’s outcomes, but their fixes and breaks nearly cancel. These comparisons are between whole systems.
  • Novelty relative to AFlow. AFlow already supports conditional single-step edits. Our contribution is the explicit patch interface coupled with paired fix/break screening under a mean-call budget.
  • Statistical evidence. We use question- or story-clustered intervals for the main comparisons. The sign test used during search is a screening heuristic. We also corrected several reported values and recomputed the main average from unrounded per-item scores, giving 84.17 rather than 84.18.

2. What the revised evidence supports

FindingEvidence and scope
Whether extra computation helps depends on both where it runs and what it executes.Under non-interference and π(r+b)>0\pi(r+b)>0, Proposition 1 gives the criterion ϵg>b/(r+b)\epsilon_g>b/(r+b). The same LiveCodeBench critique helps on disagreements and hurts on agreements. Table 33 measures the terms from paired outcomes on the main backbone.
The learned sub-protocol matters in a controlled case.The LiveCodeBench critique improves accuracy by +3.2[0.2,6.3]+3.2[0.2,6.3] points over a generic extra solve under the same trigger and call count.
The average AFlow advantage survives the clustered reanalysis.The eight-benchmark margin is +1.61[0.79,2.46]+1.61[0.79,2.46] points, with three benchmark wins and five ties. The intervals condition on the reported search.
ReProt can operate at a lower call count under AFlow’s budget.The separate budget-matched search reaches 83.59 at 2.6 calls. The main configuration has higher online call use. The comparison measures executor calls and tokens.

3. What we ask

We would appreciate your assessment of whether Proposition 1 and the paired-outcome analysis address the concern about the central claim, and whether the added controls and narrower wording address the attribution and evaluation concerns. If the revision changes your assessment, we would be grateful if you could reflect that in your review.

Thank you for your time and for helping us sharpen the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.