acceptodds
Withdrawn from ICLR 2027

Fine-Grained Credit Assignment for Agent Architecture Search

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We address the problem of agent architecture search, which constructs query-specific agent architectures from a set of candidate operators. Existing approaches optimize a controller using an architecture-level reward or advantage, assigning the same learning signal to all operator decisions within a sampled architecture, regardless of their individual contributions to the final outcome. Such coarse credit assignment can reinforce ineffective or redundant operator decisions. To address this, we present fine-grained credit assignment (FCA), which augments the architecture-level advantage with operation-level advantages for individual operator decisions. To this end, we propose state-conditioned relative advantage (SRA), which assigns operation-level rewards according to intermediate correctness and computational cost, and compares them under similar reasoning states, while placing larger weights on more similar states. We further introduce operator distribution projection (ODP), which maintains the diverse operator choices required for informative relative comparisons. Extensive experiments on diverse benchmarks using Qwen3-8B and GPT-4o-mini demonstrate the effectiveness and efficiency of FCA. We will release our code upon acceptance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agentic AI, Agentic Architecture Search, Multi-Agent Systems, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_MxJoOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FCA for training the operator-selection controller in multi-agent architecture search. During training, FCA uses both architecture-level and operation-level advantages: it computes operation rewards from the correctness of intermediate answers and token costs, then estimates operation-level advantages by comparing rewards under similar reasoning states. The authors also use operator distribution projection to maintain sampling diversity. The experiments use Qwen3-8B and GPT-4o-mini and cover MATH, GSM8K, HumanEval, MBPP, and BBH. Compared with MaAS and RAAS, FCA achieves better task performance and consumes fewer inference tokens in most settings.

Critical strengths
  1. The experiments show positive results for both performance and inference costs. With both models, task performance on all five benchmarks exceeds that of the authors' reproductions of MaAS and RAAS, while fewer inference tokens are consumed in most settings.

  2. The training analysis on BBH gives a fairly direct illustration of the credit assignment problem. By tracking the answers at each step, the authors find that the training signal for the whole architecture may not match the effect of a specific operation. They also show cases where computation continues after a correct answer is obtained and where a later operation turns a correct answer into an incorrect one. These observations show how redundant computation and answer degradation occur and explain why operation-level feedback is needed.

Critical weaknesses
  1. Necessity and novelty of the research, and the independent contribution of the core mechanism

    An important role of early multi-agent workflows of this kind, which split a long reasoning process into multiple stages, was to alleviate limitations in models' context capacity and reasoning ability. However, as model capabilities improve, context windows expand, and latent chain-of-thought techniques develop, the need for splitting tasks into external stages requires renewed justification. In this paper, the authors evaluate only two models and use short, linear workflows. I think this is insufficient to show that the framework still offers clear additional value with stronger models. In addition, the paper retains the existing operator-selection controller and search space, with its main improvements focusing on operation-level rewards, state-conditioned advantage estimation, and exploration control. I view these designs as closer to incremental optimizations of an existing framework.

    The “+SRA” setting in Table 3 simultaneously introduces operation-level rewards, the reward rules for the first correct answer and early stopping, discounting and operation-level cost penalties, and a reference reward based on state similarity. Therefore, the current ablation cannot distinguish whether the improvements in performance and reductions in inference cost come from the operation-level rewards themselves or from SRA's state-conditioned comparison. Although Table 5 in the appendix compares soft weighting with hard grouping, both use state similarity and cannot replace a control that does not use state information. Without this control, it is currently not possible to determine whether state-conditioned comparison provides clear additional value over a simpler operation-level reward mechanism.

  2. Full training costs, the cost-benefit trade-off, and the statistical reliability of performance improvements

    Regarding training costs, although the paper reports the local computational overhead of components such as SRA and ODP in the appendix, it does not adequately report the total token consumption and wall-clock time for complete training on each benchmark, and it lacks training-cost comparisons with MaAS and RAAS under comparable conditions. Another point to note is that, when recording wall-clock time, the number of concurrent calls to the model API needs to be confirmed. This is very important for a fair comparison of the measurements across different methods. The authors should include training-stage costs, test performance improvements, and reductions in inference costs in an overall evaluation to explain whether the method offers a reasonable cost-benefit trade-off at different scales of use. This information is currently missing, so it is not yet possible to judge whether the benefits of the fine-grained reward and advantage estimation mechanisms justify the corresponding training costs.

    The main experiments report only a single aggregate performance value, without variance or other statistical data, and the overall performance improvements are not large in many settings. I am concerned that these improvements could come from random fluctuations caused by uncertainty in model outputs, rather than stable gains from the method itself.

Questions for authors
  1. Please explain the additional value of this framework with stronger models and what the substantive novelty of this paper is relative to existing stepwise credit assignment and exploration methods.

  2. Although the main text and appendix specify the number of training epochs and that sampling is performed eight times per question to construct training signals, the description of how the training data are organized into batches and how parameters are updated remains insufficient. I think the authors need to clarify the following information: when training on a given benchmark, how many optimization steps does each epoch contain? How many training questions are used in each step/batch? After sampling eight times per question, how are the samples organized into batches for optimization? How many parameter updates are performed per batch, and are the sampled results reused?

  3. Please provide the total token consumption and wall-clock time for complete training on each benchmark, along with training-cost comparisons with MaAS and RAAS under comparable conditions. Also specify the number of concurrent calls to the model API when recording wall-clock time to ensure a fair comparison across methods.

  4. Please provide the variance or other uncertainty measures for the performance results to further assess whether these improvements are stable gains from the method itself.

  5. While keeping operation-level rewards, costs, sampling, and optimization settings unchanged, please compare SRA with a reference reward computed without state similarity, such as uniformly weighting the operation rewards from other architectures, to assess whether state-conditioned comparison provides additional value.

Reviewer_7imLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies credit assignment in agent architecture search. Existing methods use the final outcome to train all operator choices, even though some steps may be useful while others are redundant or harmful. The authors propose FCA, which assigns step-level advantages by comparing operator rewards across similar intermediate reasoning states. They also introduce a projection method to maintain diversity in operator selection during training.

The authors show improved accuracy and lower token usage compared to baselines such as MaAS and RAAS.

Critical strengths
  1. The paper identifies a meaningful limitation of existing agent architecture search methods: which is assigning the same credit to every operation can reinforce unnecessary steps or penalize useful ones. The analysis of credit assignment errors and redundant reasoning provides good motivation for addressing this problem.

  2. FCA improves performance across five benchmarks and two backbone LLMs, often while substantially reducing token usage. The ablations also show that both SRA and ODP contribute to these gains. The improvements in efficiency are particularly relevant given the computational cost of multi-agent systems.

Critical weaknesses
  1. The idea of combining trajectory-level and step-level advantages is closely related to GiGPO. From what I understand, the main difference is the use of semantic similarity instead of using an exact state matching which is a reasonable extension, but the novelty seems incremental. Related work such as Agent Lightning and AgentPRM is also missing from the discussion, making it difficult to assess the contribution relative to existing credit assignment methods.

  2. The operation-level reward is based on a heuristic that rewards steps leading up to the first correct answer and penalizes subsequent computation. It is unclear whether the improvements come from better credit assignment or simply from this reward design. The paper would benefit from comparisons with alternative step-level rewards and advantage estimators under the same experimental setup.

  3. SRA assumes that intermediate reasoning states with similar text embeddings provide meaningful reference points for credit assignment. However, semantically similar reasoning traces can differ significantly in correctness or the steps needed to reach a solution. The paper uses a frozen MiniLM encoder without evaluating how well its similarity scores reflect the actual reasoning states. Comparisons with alternative embeddings or correctness-aware state representations would help validate this assumption.

  4. ODP is essentially a projection that moves the operator distribution toward uniformity when it becomes too concentrated. While the paper compares it against fixed ε-greedy exploration, stronger alternatives such as entropy regularization or adaptive exploration are not considered. It is therefore unclear whether ODP offers a meaningful advantage over standard exploration techniques.

  5. The experiments use a fixed set of operators arranged in sequential architectures of at most four layers. Real agent systems can involve branching workflows, parallel execution, tool interactions, and more complex dependencies between agents. It remains unclear whether the proposed method would generalize to these settings, where identifying the contribution of individual operations may be substantially harder.

Reference:

  1. Luo, X., Zhang, Y., He, Z., Wang, Z., Zhao, S., Li, D., Qiu, L.K. and Yang, Y., 2025. Agent lightning: Train any ai agents with reinforcement learning. arXiv preprint arXiv:2508.03680.
  2. Xi, Z., Liao, C., Li, G., Zhang, Z., Chen, W., Wang, B., Jin, S., Zhou, Y., Guan, J., Wu, W. and Ji, T., 2026, April. Agentprm: Process reward models for llm agents via step-wise promise and progress. In Proceedings of the ACM Web Conference 2026 (pp. 4184-4195).
Questions for authors
  1. SRA compares operations from trajectories with similar intermediate states, but these trajectories may differ in reasoning progress and difficulty. How can we tell whether the estimated advantage reflects the operator's contribution rather than these differences? Could the authors validate SRA against counterfactual comparisons where different operators are applied to the same state?

  2. The controller selects operators based on the query and previous operator choices, while SRA uses intermediate reasoning outputs to assign credit. Why are these intermediate outputs not also used by the controller when selecting the next operator?

  3. Equation (4) assigns credit based on the first correct intermediate answer, but proximity to a correct answer does not necessarily indicate an operator's contribution. How sensitive are the results to this assumption? Would the gains hold with alternative step-level reward formulations?

  4. How much of the reduction in token usage comes from learning to stop earlier rather than selecting better reasoning operators? Could the authors separate these effects, for example by comparing accuracy at matched token budgets?

  5. Could the authors clarify whether the importance ratio in Eq. (15) gives an unbiased policy-gradient estimate, given that the operation-level advantages are computed using trajectories sampled under the projected distribution?

Reviewer_9qqJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FCA improves credit assignment for query-specific agent architecture search. It combines operation-level rewards with a state-conditioned baseline that compares similar states across sampled architectures. An operator-distribution projection mechanism maintains exploration when sampling becomes too concentrated. Experiments on five reasoning and code benchmarks with Qwen3-8B and GPT-4o-mini report improved accuracy–token trade-offs relative to reproduced MaAS and RAAS baselines, supported by ablations and overhead analyses.

Critical strengths

S1. The results cover five benchmarks and two backbones, using the same operator space for the compared search methods. On Qwen3-8B BBH, accuracy improves from RAAS's 76.5% to 81.2%, while tokens per query fall from 1,378 to 706.

S2. ODP is simple and only changes the distribution when it becomes too concentrated. The comparisons with hard grouping and fixed epsilon exploration are useful, and the reported extra training cost is small. These components also add no test-time module calls.

Critical weaknesses

W1. Table 3 adds intermediate rewards, the first-correct discount rule, operation costs, and the state-conditioned baseline together. It therefore does not show how much comes from state matching itself. A local-reward baseline without similarity weighting, plus an ODP-only control, would make the contribution clearer.

W2. Figure 1 defines useful or harmful operations through changes in intermediate-answer correctness and counts continued processing after a correct answer as over-processing. These diagnostics are closely aligned with FCA's reward heuristic, so improved curves do not independently establish faithful credit assignment. Later verification can still be useful, and MiniLM similarity may merge logically different states. More direct checks of operator utility are needed.

W3. The experiments use Qwen3-8B and GPT-4o-mini. They establish gains over the reproduced search methods on these backbones, but leave the practical case for architecture search unclear relative to a strong contemporary reasoning model. Token counts within one backbone do not show whether the searched agent is preferable in accuracy, monetary cost, or latency to a stronger single-agent baseline.

Questions for authors

Q1. I would be interested to see SRA compared with no baseline and an unweighted or query-conditioned cross-architecture baseline, while keeping the operation rewards and cost treatment fixed. What does ODP alone achieve? That would make it easier to separate the roles of intermediate supervision, similarity weighting, and exploration.

Q2. For matched reasoning states and operator budgets, do SRA credits predict downstream task benefit better than a simple local-reward baseline? In particular, what happens when a first correct answer still benefits from verification or when later operations recover from a regression? A controlled diagnostic would be more informative here than additional training-curve comparisons.

Q3. Please specify the held-out sizes and exact train/test split identifiers for the AFlow-derived datasets and BBH, the controller optimization and decoding settings, and whether intermediate ground-truth checks occur only during training. These details would help establish the evaluation protocol and reproduce the reported comparison.

Q4. Is Equation 15 intended as a local, per-state correction for a surrogate loss, or as an importance correction for trajectories under the unprojected controller? The latter interpretation also involves the distribution of earlier operator choices and future-dependent rewards. Please clarify the optimized objective and what bias, if any, is introduced by using only the current-decision ratio and averaging by sampled architecture length.

Q5. Could you compare FCA with a representative strong reasoning-capable API model as a single-agent baseline and, where feasible, as the backbone under the same operator library? A performance–cost comparison, including latency and amortized controller-training/search costs, would help distinguish the benefit of architecture optimization from simply using a more capable model.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.