acceptodds
Withdrawn from ICLR 2027

SOME-Flow: Utility-guided Structural Agentic Inference

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

As agentic reasoning shifts from static workflows toward query-adaptive topologies, reasoning structures can now adapt dynamically to individual queries. Yet structural adaptivity alone does not ensure that the resulting computation achieves favorable accuracy–cost utility, nor does it determine whether further computation remains worthwhile after an initial execution. Efficient structural inference therefore requires two coupled decisions: *how to allocate high-utility structural computation to each query initially*, and *whether additional structural computation is worth its cost after the initial execution*. To address these decisions, we introduce **SOME-Flow** (**S**tructural **O**ptimization, **M**atching, and **E**volution; hereafter **SOME**), which constructs a reusable, utility-shaped structural action space through offline topology discovery and amortizes per-query structural optimization into lightweight cost-aware matching. After initial execution, SOME selectively evolves the active structure according to predicted marginal utility, allocating additional computation only when it is worthwhile. Across five reasoning and code-generation benchmarks, SOME achieves the highest macro-average performance of 82.57%, outperforming the best automated structural baseline by 2.76 percentage points. Notably, initial structural matching alone reaches 81.41%, only 1.16 points below the full method, while reducing relative inference cost by 43.2%; selective post-execution evolution then allocates additional computation to obtain the remaining performance gains. The code is available at https://anonymous.4open.science/r/SOME-EC4B.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language models, agentic workflows, adaptive inference, multi-agent systems

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6rw6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes SOME-Flow, a utility-guided framework for query-adaptive structural agentic inference. It combines offline topology discovery, cost-aware topology matching, and selective post-execution evolution to balance accuracy and inference cost.

The authors evaluate SOME-Flow on five reasoning and code-generation benchmarks. Using Qwen3-8B as the primary executor, SOME achieves an average accuracy of 82.57%, outperforming the strongest automated baseline by 2.76 percentage points. The paper also includes ablation studies, accuracy–cost analyses, and cross-model transfer experiments.

Critical strengths

A useful perspective on utility-aware structural inference. The paper extends query-adaptive topology selection by explicitly considering whether additional computation is beneficial after initial execution. This provides a reasonable formulation for balancing performance and inference cost across different execution stages.

Systematic evaluation of the proposed components. The paper includes component-level ablations, accuracy–cost analyses, and cross-model transfer experiments. These experiments provide useful evidence for understanding the respective contributions of initial topology matching and selective continuation.

Critical weaknesses
  1. Limited evaluation on genuinely agentic tasks. All five benchmarks primarily involve single-question answering, mathematical reasoning, or function-level code generation. None evaluates long-horizon planning, external tool interactions, or dynamic environmental feedback. While these benchmarks demonstrate the effectiveness of structured multi-LLM reasoning, they provide limited evidence for the broader claims about adaptive agentic inference. This is a significant limitation, as the proposed structural adaptation may be particularly relevant in more complex, multi-step agentic scenarios.

  2. Insufficient evidence that the additional framework complexity is necessary. SOME combines offline topology search, a learned matcher, continuation prediction, and answer resolution, yet lacks comparisons against simpler alternatives, such as cost-aware routing with adaptive verification and answer resolution. Notably, full SOME improves over SOME-G1 by only 1.16 percentage points, while increasing relative inference cost from 11.04× to 19.43× (Table 1). Moreover, Table 2 lacks comparable inference costs for competing methods across executor models, making it difficult to assess the cross-model efficiency gains. Without stronger evidence against simpler, cost-matched baselines, it remains unclear whether the additional mechanisms are justified by their practical benefits.

Questions for authors
  • Could the authors evaluate SOME on at least one more realistic agentic benchmark, such as GAIA or WebArena, involving multi-step tool use or environmental interaction? This would help establish whether the claimed benefits of structural adaptation extend beyond conventional reasoning benchmarks.

  • How does SOME compare against a simpler baseline combining cost-aware routing with adaptive verification and answer resolution, under comparable inference budgets? In addition, could the authors report the token/API costs of competing methods in Table 2 and estimate the offline topology discovery and training costs?

  • How well does the learned routing and continuation policy adapt to stronger executor models? Since Table 2 transfers the policy trained on Qwen3-8B without recalibration, it remains unclear whether the same structural decisions remain cost-effective as the underlying model becomes more capable.

Reviewer_Qoj3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SOME-Flow, an agentic inference framework combining offline topology discovery, cost-aware initial structure matching, and optional supplementary computation. It first constructs a reusable bank of reasoning structures and selects an initial structure for each query. It then predicts the joint correctness of the initial and candidate supplementary structures and incorporates estimated resolver behavior to assess the marginal utility of executing a second structure.

The paper evaluates the framework on five reasoning and code-generation benchmarks and includes transfer experiments across tasks and execution models. The full method achieves a macro-average performance of 82.57%, compared with 81.41% for SOME-G1, which executes only the initially selected structure. The ablations examine initial matching, answer resolution, and continuation policies.

Critical strengths
  1. The modeling of supplementary computation considers both benefits and risks. The method considers not only whether a second structure can succeed, but also its joint correctness with the initial structure and the possibility that resolution rescues an incorrect answer or damages a correct one. This provides a reasonably comprehensive basis for estimating the expected value of continuation. The query-level out-of-fold supervision construction also reflects attention to aligning training examples with deployment decisions.

  2. The system provides useful empirical results and relevant mechanism analyses. SOME-G1 already achieves 81.41% macro-average performance, suggesting practical value in the structure bank and initial matching pipeline. Supplementary computation improves performance by 2.25, 2.67, and 0.88 percentage points on MMLU, MATH, and MBPP, respectively. The resolution analysis in Table 16 and continuation-policy analysis in Table 17 provide useful evidence about these gains and their costs.

Critical weaknesses
  1. The overall contribution remains incremental, and the supplementary-structure stage is not sufficiently compelling as a major innovation. Query-specific structure selection closely follows existing ideas in structure routing and adaptive workflow selection. Integrating discovery, matching, and continuation has system-level value, but the combination does not yet demonstrate a substantial methodological advance.

    I regard supplementary-structure selection as the more distinctive part of the framework. However, Algorithm 2 and Appendix B.6 explicitly state that MATH/MMLU continuation decisions exclude generated-answer features. Given the stated inputs, whether to continue and which supplementary structure to execute appear determinable before executing G1. On these tasks, the stage therefore primarily predicts the value of additional computation conditional on the selected initial structure, rather than adapting structural decisions to the realized generation. This limitation does not apply to all tasks: the Code admission model does use the first generated program.

    Joint correctness prediction and resolver-aware utility estimation are reasonable design choices, but the paper does not sufficiently establish how much additional capability they provide over existing structure-selection and composition approaches. The current methodological distinctions and empirical evidence are insufficient to support a strong novelty claim.

  2. The existing experiments do not sufficiently establish the independent value of supplementary-structure selection. The authors do provide relevant controls: Table 16 holds structure selection and activation decisions fixed to evaluate final-answer resolution, while Table 17 compares continuation cost coefficients and an always-continue policy to expose accuracy–cost trade-offs. These experiments are useful and should be acknowledged.

    Nevertheless, they leave a central question unanswered: with the same initial structure and resolver, and comparable actual token budgets, does learned supplementary-structure selection outperform rerunning the initial structure or executing a fixed alternative selected on development data? The Static control in Table 15 concerns initial matching and does not substitute for this continuation-stage comparison.

    Relative to SOME-G1, the full method improves macro-average performance by 1.16 percentage points while increasing the mean benchmark-normalized inference cost from 11.04× to 19.43×, approximately 76%. There are positive gains on MMLU and MATH, but on MBPP the net improvement is three correct solutions while the token cost per problem more than doubles. GSM8K and HumanEval do not activate supplementary computation and therefore provide no direct evidence of gains from this stage.

    These findings do not show that supplementary inference is ineffective. Rather, they leave unresolved whether its gains require the proposed structure-selection mechanism. Since this stage carries an important part of the paper's novelty claim, the absence of comparisons against simple alternatives under comparable budgets materially weakens the contribution.

Questions for authors
  1. According to Algorithm 2 and Appendix B.6, MATH/MMLU structure-selection and continuation decisions appear determinable before executing G1. What additional decision-making capability does “post-execution adaptation” provide on these tasks compared with planning the structure combination in advance?

  2. Tables 16–17 establish the roles of resolution and continuation policies. With the same G1 and resolver and comparable actual token budgets, how much additional benefit does learned G2 selection provide over rerunning G1 or using a fixed alternative selected on development data?

Reviewer_axwpOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper recasts query-adaptive agentic reasoning as a stade-wise allocation of structural computation under an accuracy–cost objective, and instantiates this in a three-stage pipeline. Offline, a complexity-modulated PUCT search over an executable topology space — seeded with hand-designed role graphs, expanded by local mutation, and scored by a utility blending observed correctness, a normalized cost proxy and a structural-complexity regularizer (Eq. 2) — is screened into a small frozen bank (11 / 14 / 8 structures for MATH / MMLU / Code). At inference time a matcher, trained on dense execution labels for every bank structure on every training query, predicts per-query execution success from query and topology features and is combined with an explicit cost term to select the initial structure G1 (Eq. 5). After G1 executes, a second set of predictors estimates the joint correctness of G1 and each candidate G2, maps the four joint outcomes through an offline-estimated answer-resolution profile, and computes an expected improvement over STOP (Eq. 7); a candidate is executed only if this cost-discounted marginal utility clears a gain threshold and predicted conditional-harm risk is below a cap. The empirical claim is a five-benchmark accuracy–cost result with Qwen3-8B as the common executor: a macro-average of 82.57% at 19.43× Vanilla token cost, with the matching stage alone at 81.41% and 11.04×.

Critical strengths

S1. SOME dominates the strongest baseline in its own table on both axes, and this is not a soft comparison. In Table 1, SOME reaches 82.57% at 19.43× relative cost against LLM-Debate's 81.85% at 38.93×, and the dominance is per-benchmark rather than an artifact of averaging: SOME is both more accurate and cheaper than LLM-Debate on MMLU (81.50 vs 81.00 at 2,094 vs 6,689 tokens/query), MATH (71.40 vs 70.37 at 8,751 vs 21,647) and HumanEval (84.73 vs 83.97 at 1,731 vs 9,893), all from Table 1 read against Table 10. LLM-Debate is the empirically strongest method in the table, not a straw man, so beating it at roughly half the token cost is a real result. This is consequential because the paper's central claim is an accuracy–cost claim, and the one comparison that is hardest to explain away is the one that holds up.

S2. The empirical record is unusually auditable, including where it is unfavourable. Table 10 publishes raw mean input+output tokens per benchmark for every method, so the entire Rel. Cost column of Table 1 is checkable from the paper alone — I recomputed all nine macro-averages from Table 1 and all nine Rel. Cost entries from Table 10, and they reproduce to within rounding (e.g. my recomputed SOME macro-average is 82.568 against the reported 82.57; my recomputed relative cost is 19.431 against the reported 19.43; the claimed 43.2% cost reduction is 1 − 11.039/19.431 = 43.19%). The arithmetic of the cost proxy is likewise fully specified and checks out for every row of Tables 3–5 (Section F.2's worked example, S = 256·6·(1+0.15·2) = 1996.8, follows from Eq. 11). Crucially, the paper states its own negative results: "the full-discovery and observed-seed curves both reach 0.7366 at rollout 16" (Section D.5.1, MBPP), "Direct G2 adoption alone is worse than retaining G1 on MMLU and MATH and unchanged on MBPP" (Section D.5.3), and the statement in Table 13/D.4 that GSM8K and HumanEval never activate continuation. This is consequential because it means the weaknesses below are properties of the study design rather than of unverifiable reporting — the paper gives a reader the material to find them, which is more than most submissions do and is the reason my score is a weak rejection rather than a clear one.

Critical weaknesses

W1. The gain attributed to post-execution "structural evolution" is produced by the answer resolver, not by the second structure, and the decisive control is missing. (a) Section D.5.3 holds the activation decisions and the selected G2 structures fixed and varies only how the executed outputs are used. The result is that the extra structure hurts: on MATH, "Adopt G2" gives 68.11% against 68.72% for G1 alone, and only the resolver's 71.40% exceeds G1 (Table 16). The authors state this directly — "Direct G2 adoption alone is worse than retaining G1 on MMLU and MATH and unchanged on MBPP" — and then quantify the source of the gain: "Resolution adds 10, 16, and 3 net correct answers on MMLU, MATH, and MBPP, respectively." On MATH the structural decision therefore contributes −3 answers while the resolver contributes +16. (b) The resolver is not a selector: its prompt instructs it to "discard both and solve the original problem independently" when neither candidate is supported (Section E.2), so "resolution" can be a fresh full inference. The paper does account for its tokens in the cost (Section D.3), but this means the mechanism being demonstrated is "G1, then a second opinion that may re-solve", not "a complementary topology whose marginal utility was predicted". (c) The control that would isolate the second structure — G1 followed by the resolver with no G2 — is never reported, and because Adopt-G2 is at or below G1 on all three benchmarks it is entirely possible that this control matches 71.40% at lower cost. The framing aggravates this: λ2 = 0 for MMLU and Code (Table 9), so the "cost-aware" continuation stage is cost-aware on one of the three benchmarks where it runs, and on MMLU it activates on 349/400 queries (Table 16), which is not selective. This is consequential because this stage is one of three claimed contributions and carries the entire 1.16-point gap between SOME-G1 and full SOME (82.57 − 81.41); if the second structure is not what produces that gap, the paper's central mechanism claim is unsupported by its own evidence.

W2. The cost accounting omits the dominant expenditure, and the statistical resolution is finer than the certainty the claims require. (a) Section D.3 excludes offline spend with one sentence and no magnitude: "Offline discovery, dense execution, fitting, and calibration are separate expenditures." The visible inputs are large — Table 7 lists 1,300 MMLU discovery queries at 16 rollouts each, Section D.1 has MATH starting "from dense executions on 3,000 training problems" over an 11-structure bank, and MMLU's continuation target alone is "2,535 queries and 32,955 ordered first-structure/alternative pairs" — so the supervision cost plausibly exceeds the reported inference cost by two to three orders of magnitude. A Pareto frontier (Figure 3) that omits it cannot support "its performance gains are not simply obtained through increased inference expenditure" (Section 5.2), nor answer the practitioner's question of when the bank amortizes. (b) There is one run per method, no confidence intervals, no significance tests, and only "Results describe selected checkpoints and saved execution realizations" (Section D.2), with matcher seeds "selected" (2 for MATH, 3 for MMLU and Code) and no selection criterion stated. At the reported sizes the margins are items, not percentages: HumanEval n = 131, so "improves on AFlow by 0.76 points for 8.58% more tokens" (Section D.4) is exactly one problem (111/131 vs 110/131); MBPP n = 341 makes +0.88 pp three problems; MMLU n = 400 makes the +2.25 pp continuation gain nine questions; MATH n = 486 makes +2.67 pp thirteen. (c) Two baselines report numerically identical accuracy on two benchmarks — LLM-Debate and AFlow are both 92.98 on GSM8K (= 981/1055) and both 83.97 on HumanEval (= 110/131) in Tables 1 and 11 — which is either a striking coincidence or a transcription error and must be stated explicitly; and the headline comparison is drawn against the weaker class of baseline ("outperforming the best automated structural baseline by 2.76 percentage points", Abstract) when the strongest baseline is 0.72 behind. This is consequential because the paper's claim is about efficiency, and the two quantities that would establish efficiency — total cost and the uncertainty on a small margin — are respectively unreported and unmeasured.

(Deliberately not listed as critical, but relevant to the recommendation: (i) "empirical Pareto-efficient" is a non-dominance claim within a nine-method set and is thin on two of five benchmarks — MBPP SOME uses 8336.77 tokens/query (58.5× Vanilla) against AFlow's 1343.75 (9.4×), and SOME-G1 at 3887.56 costs more than MasRouter's 3169.40 while scoring lower, 81.82 vs 82.11 (Tables 1 and 10); on GSM8K SOME gets 92.51 at 2323.68 against CoT's 92.23 at 399.43. (ii) Eq. 1 posits a single λ but the deployed settings are λ1 = 0.25/0.15/2 and λ2 = 0.005/0/0 (Tables 9, 18), so the macro-average aggregates three different cost preferences; Table 17 further shows the deployed MATH point (71.40 at 1354.79) is accuracy-inferior to λ2 = 0 (72.84 at 1935.50), i.e. it is a cost choice rather than an optimum. (iii) Discovery shows no benefit over its seeds on MBPP — "0.7366 seed / 0.7366 full" (Table 14), confirmed by the Figure 6 caption "Full and observed-seed curves coincide" — over a 90-query search, yet the HumanEval result is transferred from that same MBPP bank. (iv) MaAS, "Multi-agent Architecture Search via Agentic Supernet" (arXiv:2502.04180, ICML 2025 oral), learns a query-conditional distribution over architectures that samples a different multi-agent system per query under a performance–cost objective, with early-exit operators; it is the closest prior art to "a frozen bank with query-conditional selection plus post-execution early exit" and is not cited. Cost-aware routing baselines including RouteLLM and FrugalGPT are likewise absent from the reference list, though both are standard comparators in this literature — they appear as baselines in MasRouter's own Table 1 [public source: arXiv:2502.11133]. — and the third stage's framing is close to the cascade / learning-to-defer line that these papers instantiate.)

Questions for authors
  1. The missing control: G1 + resolver, no G2. On MMLU, MATH and MBPP, with the same resolver prompt and the same activated query set, what accuracy does G1 followed by the resolver achieve without executing any complementary structure? Given that "Adopt G2" is at or below G1 on all three benchmarks, this is the experiment that establishes whether the second structure contributes anything beyond what the resolver would deliver anyway. If you have the saved G1 outputs and resolution records this should be computable without new inference.

  2. Total cost, including the offline term. Please report total model calls, input+output tokens, and wall-clock or dollar cost for discovery (16 rollouts × the stated discovery sets), dense execution of the bank over the supervision sets, matcher fitting and continuation calibration, then state the break-even: after how many evaluation queries does the frozen bank pay for itself relative to the baselines you compare against? Which of these quantities were logged during the study?

  3. Variance and the two identical baseline values. Can you give ± std over at least three seed/realization repeats, or bootstrap confidence intervals, for each cell of Table 1? State the criterion by which matcher seeds 2 and 3 were "selected". Confirm that the identical LLM-Debate and AFlow values (92.98 on GSM8K, 83.97 on HumanEval) are independent runs rather than a transcription artefact. And state explicitly that the 3,000-problem MATH supervision pool is disjoint from the 486 level-5 evaluation problems — the text asserts only that those queries were "never used to fit either the matcher or the continuation predictors" (Section F.1), not that the pools are disjoint.

  4. The λ settings and the operating point. Why λ1 ∈ {0.15, 0.25, 2} across datasets, and what does the macro-average become when all five benchmarks are scored at a common λ? On MBPP, SOME-G1 consumes 3887.56 tokens/query — more than MasRouter's 3169.40 — while scoring 81.82 against 82.11; is that an operating-point choice or a matcher-calibration failure? On HumanEval, λ1 = 2 drives composite cost to 23.47% of Static while raising accuracy from 81.68 to 84.73 (Table 15): does that mean the cheapest code structure dominates over the whole bank, and if so what is the structural matcher contributing there beyond picking the cheapest arm?

  5. Baseline protocol and a reproduction check. How were AFlow, GPT-Swarm and MasRouter configured, tuned and budgeted (search iterations, token budget, prompt set), and are the per-executor numbers in Table 2 your own runs or quoted? In particular, did your MasRouter re-implementation reproduce its published operating point before being restricted to a single executor? MasRouter's contribution is routing among a pool of LLMs, so with a single Qwen3-8B the routing dimension is degenerate; its original paper reports a five-benchmark average of 85.93 over a four-model pool [public source: arXiv:2502.11133, Table 1], against the 79.81 you report. I recognise the backbone difference makes these not directly comparable — the reason I am asking rather than asserting is that I could not establish whether the re-implementation is faithful. Relatedly, in that same published table AFlow exceeds LLM-Debate on MATH by 8.67 points (73.35 vs 64.68), whereas in your Table 1 AFlow is 7.20 points below LLM-Debate (63.17 vs 70.37); the two tables use different MATH subsets, but a sign inversion of this size deserves an explanation or a sanity check against published numbers.

  6. Discovery's contribution on the code family. MBPP discovery produces no improvement over the seeds (0.7366 vs 0.7366 over 90 queries, Table 14). Given that HumanEval's entire result is obtained by transferring that MBPP bank, how much of the MBPP and HumanEval performance survives using only the seven seed structures? And what does this imply for the claim in Section 5.3 that discovery "does more than expand the candidate pool" — is that claim meant to hold for all three task families or only for MATH?

Criteria under which my score would rise: (1) a G1 + resolver-without-G2 control on the three evolution-active benchmarks, showing that the complementary structure adds accuracy beyond the resolver; (2) total-cost accounting that includes discovery, dense execution and calibration, plus a break-even statement; (3) multi-seed or bootstrap uncertainty on Table 1, and an explanation of the two identical baseline values; (4) the λ settings justified or replaced by a single reported frontier. Meeting (1) and (2) would move me to 3 (weak acceptance); meeting (1)–(3) together with a reconciled λ story would make this a 4, since the underlying accuracy–cost trade-off against the strongest baseline (S1) is genuinely good. Conversely, if (1) shows that G1 + resolver already reaches 71.40% on MATH, the third stage should be withdrawn as a contribution and the paper's centre of gravity moves to the bank-plus-matcher design, which I would then judge on its own — plausibly below the current score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.