acceptodds
Under review as a conference paper at ICLR 2027

From Belief To Action: A Bayesian Co-Evolution Framework For LLM Agent System

Abstract

LLM-centric agent systems have shown remarkable potential, yet their self-evolution from experience remains limited. Existing approaches either optimize context alone while keeping the agent model fixed, or perform joint adaptation for them across heterogeneous spaces. In both cases, nevertheless, context and model are treated as decoupled entities, lacking a common ground for representing and propagating their mutual uncertainties. We propose a unified Bayesian inference framework that treats context and agent model as coupled variables within a single probability measure space, directly updated by online trajectory likelihoods without parameter retraining. This enables few-shot trial-and-error adaptation, where belief updates for the variables are derived directly from empirical returns. Experiments demonstrate consistent outperformance over both existing paradigms, substantiating the viability of formulating self-evolution as principled inference under coherent probabilistic belief dynamics.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
LLM-centric agent, self-evolution, joint adaptation, Bayesian inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_EtbnOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a Bayesian inference framework to simultaneously infer context variables and agent parameters which in this case is the expected reward θ\theta at inference time. Observed trajectory return is used as evidence to update the beliefs and sampled reward belief is used to design an advantage function to modify the action distribution, essentially control exploration vs exploitation. Experimental results on DeepSearch and τ2\tau^2 bench show strong performance compared to baselines and improvement over iterations.

Critical strengths
  • The experimental results clearly demonstrate the effectiveness of the proposed method.
  • The Bayesian update mechanism is sound.
Critical weaknesses
  • The paper is difficult to follow. The authors heavily used abstract probabilistic modeling language to describe what seems to be a harness engineering problem. Especially for readers not familiar with the task environment, it's hard to understand what context and agent parameters represent.
  • The beliefs seem to affect action selection only via the advantage function which essentially performs temperature tuning. So the level of control over action distribution seems limited.
Questions for authors

NA

1 comment

Dear Reviewer,

Thank you again for your great work on reviewing this paper submission to ICLR 2027.

The authors have now submitted their rebuttal. The open reviewer-author discussion phase will end soon, on November 18 at 11:59 PM AoE.

Please read the authors' responses, determine whether any further clarification is needed, and respond to the rebuttal at your earliest convenience. Please be sure to acknowledge the authors' responses and update your review accordingly.

Best regards, Your AC

Reviewer_wLGoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a Bayesian co-evolution framework for the test-time adaptation of LLM-based agent systems. The authors argue that existing approaches either optimize external contexts while keeping the underlying model fixed or jointly optimize contexts and model parameters through alternating procedures, without explicitly maintaining their coupled uncertainties.

To address this limitation, the paper introduces two probabilistic variables: a categorical context variable ϕ\phi, which controls the selection of context modification strategies, and a continuous agent variable θ\theta, which represents an estimate of the expected trajectory return. The framework iteratively samples these variables, modifies the context, adjusts the agent's action probabilities through a reward-guided mechanism, and updates the variable distributions using observed trajectory returns.

The authors further derive Bayesian update rules and claim theoretical guarantees of monotonically improving trajectory returns. Experiments on DeepSearch and$\tau^2$-Bench, using multiple backbone models, report improvements over context optimization and alternating joint optimization baselines. Ablation studies are provided to assess the contributions of the two variables.

Critical strengths

1. A potentially useful approach to parameter-free joint adaptation.

The paper explores a practically relevant direction by combining context modification with action-level adaptation without updating the underlying LLM parameters. The categorical representation of context editing operations and the return-guided adjustment of action probabilities provide a relatively straightforward framework for test-time adaptation. This is potentially valuable in settings where repeated parameter fine-tuning is expensive or impractical.

2. Encouraging empirical results across multiple agent settings.

The evaluation covers two agent benchmarks and multiple backbone models. The reported improvements are particularly noticeable on τ2\tau^2-Bench, where the proposed method achieves higher multi-run success rates than the compared baselines. The component-level ablations also suggest that combining context adaptation and action-level control may be more effective than applying either component independently. These findings motivate further investigation of the proposed approach.

Critical weaknesses

1. Substantial concerns regarding the consistency of the probabilistic formulation and the validity of the theoretical guarantees.

My primary concern is that the proposed Bayesian formulation does not appear to be internally consistent, which directly affects one of the paper's central contributions.

First, Eq. (4) defines the conditional distribution of the agent variable as p(θ∣ϕ=χi)=N(μ+δi,σ2)p(\theta\mid\phi=\chi_i)=\mathcal{N}(\mu+\delta_i,\sigma^2). However, Algorithm 1 (Line 2) samples ϕ\phi and θ\theta separately, with θ∼N(μ,σ2)\theta\sim\mathcal{N}(\mu,\sigma^2). It is therefore unclear whether the actual implementation follows the conditional generative model assumed in the theoretical derivation. In particular, Lemma 1 relies on the conditional dependence between these variables to obtain a trajectory-return distribution with mean μ+2δi\mu+2\delta_i.

Second, the derivation of Lemma 2 appears problematic. Marginalizing the context variable from Eq. (6) generally yields a Gaussian mixture rather than the single Gaussian N(θ,στ2)\mathcal{N}(\theta,\sigma_\tau^2) claimed in the paper. The mixture weights should account for the dependence between ϕ\phi and θ\theta specified in Eq. (4). Even under an independent-sampling approximation, the additional variance induced by the context-dependent offsets does not generally disappear. Consequently, Eq. (8), while algebraically correct for a Gaussian prior and Gaussian likelihood, is not established as the exact Bayesian posterior under the originally specified joint model.

More importantly, Theorem 2 claims that E[vτk−θk]>0\mathbb{E}[v_{\tau_k}-\theta_k]>0 and that the expected agent variable increases monotonically across iterations. However, under the joint model defined by Eqs. (4) and (6), the stated constraint ∑iωiδi=0\sum_i\omega_i\delta_i=0 implies E[vτ−θ]=0\mathbb{E}[v_\tau-\theta]=0, which is inconsistent with the claimed strict inequality. Furthermore, the proof in Appendix A.4 appears to rely on selecting the coefficient κ\kappato ensure sufficient action rewards without establishing that such rewards are achievable in the underlying environment. A Bayesian posterior update also does not, by itself, guarantee monotonic improvement in the posterior mean or actual task return.

These issues substantially weaken the paper's claims of a unified Bayesian inference framework with rigorous monotonic improvement guarantees. Clarifying the underlying generative model, distinguishing exact inference from approximation, and establishing the necessary assumptions for the theoretical results would be essential.

2. Insufficient experimental evidence to isolate the contribution of Bayesian co-evolution from additional test-time computation and reward-guided action selection.

The proposed method performs multiple evolution iterations, generates multiple trajectories per iteration, and considers multiple candidate actions whose probabilities are subsequently adjusted. This introduces additional inference and interaction costs. However, the paper does not sufficiently establish whether the reported improvements arise specifically from the proposed Bayesian belief updates or primarily from increased test-time computation.

Although the ablation study compares the full method against context-only and agent-variable-only optimization, it does not isolate the contribution of Bayesian inference itself. Comparisons against simpler alternatives, such as budget-matched repeated sampling, reward-guided candidate reranking, or heuristic context-selection updates, would be particularly informative.

Moreover, the continuous variable θ\theta does not represent an updated LLM policy parameter; instead, it controls action selection through a return-dependent probability transformation. This raises an important question about the distinction between the proposed agent-model co-evolution mechanism and existing test-time sampling or reranking strategies.

The paper would benefit from clearer reporting of the number of sampled trajectories, candidate actions, total model calls, token consumption, and tool interactions, together with matched-budget comparisons. Without these controls, it remains difficult to attribute the performance gains specifically to the proposed coupled Bayesian formulation rather than to additional exploration and computation.

Overall, these concerns affect both the theoretical soundness and the empirical justification of the paper's principal contribution.

Questions for authors

Q1. Consistency of the joint probabilistic model and posterior updates.

Could the authors clarify whether θ\theta is sampled conditionally on the selected context variable ϕ\phi, as defined in Eq. (4), or independently, as written in Algorithm 1? In addition, how is the Gaussian marginal likelihood in Lemma 2 obtained from the original joint model? Are Eqs. (7) and (8) intended to represent exact Bayesian posteriors or approximations? If approximations are used, please clarify their assumptions and potential impact on the updates.

Q2. Validity and assumptions of Theorem 2.

How is the claimed inequality E[vτk−θk]>0\mathbb{E}[v_{\tau_k}-\theta_k]>0 reconciled with Eqs. (4) and (6), which appear to imply a zero unconditional expected difference under the stated constraint ∑iωiδi=0\sum_i\omega_i\delta_i=0? Furthermore, what assumptions guarantee that adjusting κ\kappa can produce actions with sufficiently high rewards? A precise statement of the necessary conditions, or a corrected theoretical guarantee, would substantially affect my assessment of the paper's soundness.

Q3. Attribution of performance gains and computational efficiency.

Could the authors provide the exact numbers of evolution iterations, trajectories per iteration, and candidate actions, together with total inference-token and tool-call costs? How would the proposed method perform against simpler test-time alternatives, such as reward-guided reranking or repeated sampling, under matched computational budgets? Clarifying how rewards are obtained for candidate actions before final selection would also help establish the practical implementation of Eq. (5).

1 comment

Dear Reviewer,

Thank you again for your great work on reviewing this paper submission to ICLR 2027.

The authors have now submitted their rebuttal. The open reviewer-author discussion phase will end soon, on November 18 at 11:59 PM AoE.

Please read the authors' responses, determine whether any further clarification is needed, and respond to the rebuttal at your earliest convenience. Please be sure to acknowledge the authors' responses and update your review accordingly.

Best regards, Your AC

Reviewer_a9PWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a Bayesian co-evolution framework for LLM agents, enabling joint adaptation of context modification and action generation without updating model parameters. The framework introduces probabilistic variables for context selection and expected trajectory returns, which are updated through online interaction feedback. Experiments on DeepSearch and τ²-Bench demonstrate promising improvements over existing baselines.

Critical strengths
  1. The paper addresses an interesting and relevant problem in online LLM agent adaptation, with a unified probabilistic perspective.
  2. The proposed framework is relatively simple and avoids expensive model fine-tuning, making it potentially useful for practical applications.
  3. The experimental results are encouraging, especially the consistent improvements on τ²-Bench across different backbone models.
Critical weaknesses
  1. The monotonic improvement guarantee in Theorem 2 appears to rely on assumptions that are not sufficiently clarified. In particular, the Bayesian posterior mean does not necessarily increase when the observed return is below the prior expectation. Further clarification of the theoretical conditions would strengthen the paper.

  2. The proposed framework models context and agent variables jointly, but the inference procedure involves several simplifying approximations. A more detailed discussion of their validity and potential impact would be helpful.

  3. Although the results are promising, comparisons under matched inference budgets and additional statistical analysis would make the conclusions more convincing. Furthermore, comparisons with simpler alternatives, such as adaptive temperature sampling and reward-based reranking, would help clarify the contribution of the proposed mechanism.

Questions for authors
  1. Could the authors clarify the assumptions under which Theorem 2 guarantees monotonic improvement?
  2. How sensitive is the method to the Gaussian approximation and the choice of κ?
  3. Could the authors provide additional analysis of inference costs and comparisons with simpler sampling-based methods?
1 comment

Dear Reviewer,

Thank you again for your great work on reviewing this paper submission to ICLR 2027.

The authors have now submitted their rebuttal. The open reviewer-author discussion phase will end soon, on November 18 at 11:59 PM AoE.

Please read the authors' responses, determine whether any further clarification is needed, and respond to the rebuttal at your earliest convenience. Please be sure to acknowledge the authors' responses and update your review accordingly.

Best regards, Your AC

Comments on the paper

1 comment
AuthorsOpenReview ↗

rebuttal summary

We would like to provide a complete summary of our rebuttal as the final response. We sincerely thank the reviewers and chairs for their comments and efforts.

Theoretical part

  1. Reviewers a9PW and wLGo both raised the issue of the monotonic increase of the benefit in Theorem 2. We have provided a detailed explanation. First, the monotonicity follows directly from the component control in Eq. (5) and is independent of the distribution of θ\theta itself. The distribution of vτ∣θv_\tau|\theta follows a truncated Gaussian distribution, but during Bayesian inference updates, the truncation term does not participate in updating the Gaussian core. For simplicity, we removed this truncation term, which caused some misunderstanding. In the original paper, we used the expression in Theorem 2 to maintain logical continuity: we first introduced the variable, then described its update, and finally clarified its monotonicity. In the revised paper, we have decomposed the content of Theorem 2 to make everything clearer. The new version no longer causes any misunderstanding, and it does not affect the original theoretical conclusions or the experimental validation in the paper.

  2. Regarding the mixture-of-Gaussians approximation issue, we emphasized in the rebuttal that our setting is an online agent optimization scenario, where only one feedback trajectory is obtained at each step. Our goal is to update the agent quickly from a small number of feedback trajectories and improve task success rate. A mixture-of-Gaussians formulation would be essentially intractable. Therefore, we adopt a necessary approximation to make the problem efficiently solvable, rather than aiming for a precise solution based on large amounts of data, which is infeasible. In the rebuttal, we responded to this approximation issue and emphasized that although the approximation introduces error, it is the only viable choice in this setting, and extensive experiments demonstrate its effectiveness.

  3. Regarding the difficulty in understanding the paper raised by Reviewer Etbn, we have added a high-level motivation and comparative description in the revised paper, as well as visual Figure 4 (the revised paper). Based on the reviewer’s latest feedback, this has been understood. At the same time, we emphasize that the design of Eq. (5) is advantage-guided, as shown by our theoretical proof in Section A.2. This also addresses Reviewer Etbn’s most recent question.

Experimental issues

  1. We added a reward-rerank experiment and provided a comparison of token consumption, and a detailed computational example regarding Eq. (5). This addresses the concerns of Reviewers a9PW and wLGo. Although the actual update of θ\theta is based on reward, the overall performance improvement comes from the joint effect of each module, including reward-induced probability steering and Bayesian optimization, rather than from reward alone.

  2. We added an ablation study on the sensitivity of κ\kappa, which addresses Reviewer a9PW. It demonstrates that our designed hyperparameter indeed needs to be near the theoretical lower bound to achieve good performance, which is consistent with the theoretical results.

  3. Regarding the constant-form comparison mentioned by Reviewer Etbn in the most recent comment, we emphasize that the ϕ\phi optimization ablation in our paper is exactly such a comparison. It demonstrates that the performance improvement is not merely due to a fixed advantage, but rather due to the Bayesian optimization of θ\theta.

Finally, we would like to state that our primary motivation is how to drive an agent system to improve its capabilities and successfully solve problems from a small amount of trial-and-error feedback. This differs from learning from large-scale data and is more aligned with an ideal form of agent evolution. We provide a feasible and theoretically rigorous approach, together with extensive experimental validation. Our theory makes two major contributions. First, we jointly optimize the context and the model in the agent system within a shared probability space, ensuring the consistency of feedback information. Second, and most importantly, our theory guarantees that the trajectories sampled in each interaction with the environment are not random but follow a benefit-increasing direction, so that after multiple interactions the trajectory actually executed will approach success.

We hope this work can shed light on efficient agent self-improvement from limited feedback and inspire future research in this direction.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.