acceptodds
Under review as a conference paper at ICLR 2027

Choose Your Moment: Selective Evolution of LLM Agents via Latent Reasoning Dynamics

Abstract

Long-horizon LLM agents can continually improve by consolidating accumulated experience into their model parameters, yet indiscriminate updating is neither necessary nor always beneficial. Frequent updates incur substantial training costs and may overfit transient experience or interfere with previously acquired capabilities, whereas delayed updates can force agents to repeatedly reconstruct reasoning patterns that could have been internalized. This raises a key question for self-evolving agents: when is model evolution actually worthwhile? In this paper, we study this problem through the lens of latent reasoning dynamics and introduce Latch, a selective evolution framework that encodes latent reasoning trajectories and extracts deep-to-shallow reasoning corrections. We further characterize whether these corrections are locally realizable by a shared parameter change and remain consistent across past queries, providing evidence of whether the accumulated reasoning patterns can be jointly internalized. These signals are then used to predict the finite-horizon advantage of updating immediately over waiting, supervised by paired update–wait rollouts that measure downstream task performance and capability retention. For evolution steps predicted to be worthwhile, we perform on-policy self-distillation, using a frozen deeper reasoning path to supervise the current shallow model on student-generated prefixes while using experience replay to mitigate degradation of prior capabilities. Experiments across long-horizon benchmarks demonstrate consistent improvements in cumulative task utility and capability retention with reduced update overhead.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Latent Reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_usPQOpenReview ↗
Rating
2(+1)
Confidence
4(+2)
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies when a long-horizon LLM agent should perform model parameter updates as it continuously accumulates experience, rather than adopting fixed-period update schedules or strategies based on superficial signals such as error rates. The authors propose LATCH, a selective evolution framework based on latent reasoning dynamics. This method compares deep and shallow latent reasoning paths to extract reasoning corrections, and assesses, through analyses of local parameter realizability and cross-question consistency, whether these corrections can be jointly realized by a shared parameter update. The model then uses a predictor trained under the supervision of paired Update–Wait Rollouts to estimate the future benefit of updating immediately versus waiting. When the predicted benefit is positive, LATCH triggers On-Policy Self-Distillation to distill the information from the deep reasoning path into the shallow model, combined with experience replay to mitigate capability forgetting. The experiments use two backbones and are evaluated on seven benchmarks spanning mathematical reasoning, code generation, general reasoning, and multi-hop question answering. Through comparisons with various update strategies and experience-learning methods, along with ablation studies, update-timing analysis, hyperparameter sensitivity analysis, and computational cost analysis, the authors demonstrate LATCH's advantages in both task performance and efficiency.

Critical strengths
  • The research question has practical significance. This paper focuses on when a continuously operating LLM agent should internalize its accumulated experience into model parameters, rather than merely studying how to perform parameter updates. This question involves trade-offs among update cost, adaptation speed, and the retention of existing capabilities, and is of practical research value for continual learning and self-evolving agents.

  • The experiments cover multiple reasoning tasks and base models of different scales. The authors evaluate the method on seven benchmarks and two backbones, and provide module ablations, update-timing visualizations, and computational cost analyses. These results provide some empirical support for the effectiveness of the complete framework, and also indicate that its performance improvements do not appear only on a single task.

Critical weaknesses
  • The paper lacks a convincing Motivation–Method–Evaluation argumentation loop. The two core observations lack direct empirical or theoretical validation. Specifically, (1) the assumption that Latent Reasoning Corrections capture valuable reasoning patterns remains unverified; (2) the relationship between Shared Parameter Realizability and actual Evolution Advantage is not established; and (3) the ablation studies only demonstrate performance differences without validating the proposed mechanisms or the necessity of individual modules. Overall, the framework appears to rely on heuristic design choices rather than well-validated scientific hypotheses, weakening its core technical contribution.

  • The experimental setup does not adequately support the claimed long-horizon agent setting, and critical evaluations are missing. Specifically, (1) the benchmarks primarily involve static reasoning and code generation rather than long-horizon interactive agent environments such as AppWorld or WebArena; (2) controlled comparisons against simpler feature-based predictors using Accuracy, Loss, Uncertainty, or Gradient Statistics are missing; and (3) the Evolution Advantage Predictor lacks direct evaluation of prediction accuracy, correlation with actual update benefits, and update decision quality. These gaps limit the generalizability of the findings and leave the paper's central novelty claims insufficiently supported.

Questions for authors
  1. Can the authors provide direct evidence that a stable relationship exists among Latent Reasoning Corrections, Local Realizability, Cross-query Consistency, and actual model update benefits? For example, could you report the correlation between these diagnostic metrics and the measured Update–Wait Advantage, or use controlled experiments to verify whether high realizability and high consistency indeed correspond to more favorable parameter updates? This evidence is essential for judging whether the method's design rests on a sufficient scientific basis.

  2. How much gain does LATCH's complex diagnostic mechanism actually deliver compared with an Evolution Advantage Predictor that uses simple features? Could you compare, under the same training data, prediction target, update budget, and OPSD mechanism, predictors that use Accuracy, Loss, Uncertainty, or Gradient-based Features? This comparison will directly affect my assessment of the method's necessity and technical contribution.

  3. Can LATCH remain effective in real long-horizon interactive agent environments? The current experiments are mainly based on reorganized streams of static reasoning tasks. Could you evaluate the method in AppWorld, WebArena, or similar environments, or provide sufficient evidence that the current task streams are representative of the long-horizon agent self-evolution scenario emphasized in the paper? This will directly affect my judgment of the scope of applicability of the paper's conclusions.

  4. How do the authors ensure that deeper latent reasoning paths provide reliable correction targets? The paper uses deep reasoning states as references for shallow reasoning, but deeper reasoning does not necessarily correspond to better task performance. Could the authors clarify the actual performance gap between the Deep Teacher and the Shallow Student, and how they handle cases where the deep path produces erroneous or unhelpful corrections? This concerns the soundness of the entire Latent Correction mechanism.

6 comments
AuthorsOpenReview ↗

Response to Reviewer usPQ (1/4)

We sincerely thank the reviewer for the valuable suggestions and comments. We provide the following point-by-point response.

Q1: Provide evidence that a stable relationship exists among Latent Reasoning Corrections, Local Realizability, Cross-query Consistency, and actual model update benefits.

Thank you for the constructive suggestion.

To directly validate the proposed diagnostic signals, we conduct a correlation analysis between latent correction magnitude (DD), local realizability (RR), cross-query consistency (CC), and measured Update–Wait Advantage (At,HA_{t,H}, Eq. 8). At each decision point tt, we define Dt=1∣Pt∣∑i∈Pt∥δi∥2D_t = \frac{1}{\lvert\mathcal{P}_t\rvert} \sum_{i \in \mathcal{P}_t}\lVert\delta_i\rVert_2, where δi\delta_i denotes the deep-to-shallow correction in Eq. (3), while RtR_t and CtC_t follow Eq. (9). We compute Spearman rank correlations across evolution decision points on four representative benchmarks and two backbones, using paired Update–Wait rollouts to estimate actual update benefits. The results are reported in Table A.

Table A: Spearman rank correlations between diagnostic signals and measured Update–Wait Advantage across representative benchmarks and backbones.

BenchmarkBackboneρ(D,A)\rho(D,A)ρ(R,A)\rho(R,A)ρ(C,A)\rho(C,A)
GSM8KLLaMA-3.1-8B0.2940.7560.819
GSM8KQwen3.8-27B0.2710.7820.798
HumanEval+LLaMA-3.1-8B0.3390.7240.775
HumanEval+Qwen3.8-27B0.3180.7480.762
BBHLLaMA-3.1-8B0.3510.7370.791
BBHQwen3.8-27B0.3190.7630.774
HotpotQALLaMA-3.1-8B0.2580.6920.728
HotpotQAQwen3.8-27B0.2460.7040.716

As shown in Table A, latent correction magnitude exhibits relatively weak correlations with measured evolution advantage, indicating that larger reasoning discrepancies alone do not reliably identify beneficial updates. In contrast, local realizability and cross-query consistency exhibit substantially stronger positive correlations across all evaluated settings, supporting their use in assessing the potential benefits of model evolution rather than relying on correction magnitude alone. The results have also been incorporated into Appendix E.6 of the revised manuscript (Table 8).

AuthorsOpenReview ↗

Response to Reviewer usPQ (2/4)

Q2: Compare the performance of predictors using Accuracy, Loss, Uncertainty, or Gradient-based Features under the same training data, prediction target, update budget, and OPSD mechanism.

Thank you for the thoughtful suggestion.

To isolate the contribution of LATCH's diagnostic mechanism, we additionally compare four simple-feature evolution advantage predictors: Accuracy, the mean verifier outcome; Loss, the mean token-level negative log-likelihood of recorded completions; Uncertainty, the mean predictive entropy across generated tokens; and Gradient, the mean L2 norm of adapter gradients computed from the token-level loss. All features are computed from the same recent experience window, with Loss, Uncertainty, and Gradient evaluated under the current adapter. All methods share the same training data, paired Update–Wait Advantage targets (Eq. 8), predictor architecture, update budget, and OPSD settings. The results are reported in Table B.

Table B: Predictor Features Comparison under LLaMA-3.1-8B across different tasks.

Predictor FeaturesGSM8K (EM ↑)HumanEval+ (Pass@1 ↑)BBH (Acc. ↑)HotpotQA (F1 ↑)
Accuracy81.562.861.246.5
Loss82.364.262.646.3
Uncertainty82.763.463.848.2
Gradient83.963.763.147.7
LATCH88.269.467.952.8

LATCH consistently outperforms all simple-feature predictors across the evaluated benchmarks, while the strongest individual feature varies across tasks, indicating that conventional outcome-, uncertainty-, and gradient-based signals provide limited evidence of the benefits of immediate evolution. By jointly modeling deep-to-shallow corrections, shared parameter realizability, and cross-query consistency, LATCH provides more informative evidence for selecting beneficial updates, supporting the effectiveness of its diagnostic mechanism. The results have also been incorporated into Appendix E.7 of the revised manuscript (Table 9).

AuthorsOpenReview ↗

Response to Reviewer usPQ (3/4)

Q3: Provide a performance comparison on real long-horizon interactive agent environments, such as AppWorld and WebArena.

Thank you for the valuable comment.

To evaluate LATCH beyond static reasoning streams, we conduct experiments on two interactive agent benchmarks, AppWorld and WebArena, involving multi-step API interactions and state-changing web operations, respectively. We organize tasks into sequential experience batches, allowing agents to accumulate execution trajectories and environment feedback, with LATCH selectively updating its reasoning adapter between batches. We evaluate AppWorld on Test-Normal using Task Goal Completion (TGC) and WebArena using Success Rate (SR). All methods share the same backbones, task streams, and evaluation protocols, with results averaged over three independent runs. The results are given in Table C.

Table C: Performance comparison on long-horizon interactive agent benchmarks across different backbones.

MethodsAppWorld (LLaMA-3.1-8B)AppWorld (Qwen3.8-27B)WebArena (LLaMA-3.1-8B)WebArena (Qwen3.8-27B)
Vanilla18.833.812.427.8
Periodic20.135.214.129.5
Random19.634.713.928.2
Early20.535.614.530.4
ACE25.842.418.634.5
ReMe27.445.120.536.2
MaTTS26.543.722.839.8
EvolveR28.346.321.638.4
LATCH31.950.526.742.6

LATCH consistently achieves the best performance across both interactive environments and backbones, demonstrating the effectiveness of selective evolution in tasks involving multi-step interactions and environment feedback. Our original streams model long-horizon self-evolution across successive tasks, incorporating recurring task types, changing task demands, sequential feedback, and repeated update–wait decisions. The additional interactive evaluations extend this setting to environment-state transitions and execution feedback, further supporting LATCH's applicability to continual self-evolution in both reasoning-intensive and interactive agent environments. The performance results have also been incorporated into Section 5.2 of the revised manuscript (Table 2).

Reviewer_SjS8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates when a long-horizon LLM agent should update its trainable reasoning adapter rather than continuing with its current parameters. The authors formulate update timing as a sequential decision between updating immediately and waiting, proposing the LATCH framework. LATCH achieving consistent gains across seven benchmarks on LLaMA-3.1-8B and Qwen3.8-27B.

Critical strengths
  1. Formulating model adaptation as a comparative decision between updating now and waiting over a future horizon is a well-motivated departure from standard reactive triggers. Rather than relying solely on current error rates or uncertainty proxies, supervising the decision policy with paired update-wait rollouts that explicitly penalize degradation on a retention set (Equation 8) provides a principled way to balance immediate task responsiveness against long-term stability.
  2. Decoupling the presence of a supervision signal from its local parameter-space realizability addresses a practical bottleneck in continual adaptation. By partitioning diagnostic queries into disjoint fitting and consistency subsets (Equation 7) before triggering distillation, the method explicitly tests whether query-specific corrections share compatible update directions instead of blindly forcing conflicting updates into the adapter.
Critical weaknesses
  1. The empirical evaluation does not adequately isolate the benefit of selective waiting from the underlying update operator. Although the paper argues that indiscriminate updating causes overfitting, capability interference, and excessive overhead, Table 1 and Table 2 omit the most direct baseline: updating with the exact same on-policy self-distillation and replay procedure at every decision point without waiting. Furthermore, the paper does not clearly specify in Section 5.1 or Appendix D.2 whether the timing baselines (Periodic, Random, and Early) use the exact same optimization objective and step budget as LATCH. In Figures 4 and 9, Periodic, Random, and Early perform the same number of updates as LATCH (six updates) and partially overlap with the beneficial reference points, yet in Table 1 they improve over Vanilla by less than 1.5 percentage points on GSM8K and MBPP+, whereas LATCH improves by 8.6 to 10.1 points. Without an update-every-batch baseline and a transparent verification that all timing controls share the identical update operator, it is difficult to attribute this large performance gap purely to update timing.

  2. The core supervision mechanism relies on an unverified assumption that running additional recurrent latent transitions on a frozen off-the-shelf backbone yields a reliable teacher. Standard backbones such as LLaMA-3.1-8B and Qwen3.8-27B are not pre-trained to reason via continuous hidden-state recurrence, and the paper does not explain how the initial adapter ϕ0\phi_0 is pre-trained or initialized so that iterative transitions in Equation 2 are meaningful at the start of the stream. More importantly, during online updates (Equations 10 and 11), the adapter is only optimized at the shallow depth NS=2N_S=2. When the updated adapter ϕt\phi_t is subsequently unrolled for NT=4N_T=4 steps to serve as the teacher, the additional transitions operate out-of-distribution relative to how ϕt\phi_t was trained. The paper provides no direct evaluation comparing the standalone accuracy of the NT=4N_T=4 path against the NS=2N_S=2 path over the course of the stream to confirm that the deeper path remains a superior teacher after multiple shallow updates.

  3. Several methodological details, cost accounting practices, and statistical evaluations lack rigor. In Section 4.3, the probe objective in Equation 6 combines output-level divergence and latent displacement to construct the subspace StS_t, but the first-order response GiG_i and fitting error in Equation 7 silently drop the output-level divergence term, and neither the projection dimension of Proj nor the split ratio between the fitting set and consistency set is specified. In addition, Table 2 reports that LATCH achieves lower latency (4.68s) than Periodic updating (4.84s) despite rerunning 16 diagnostic queries and computing per-query Jacobians at every decision point, while completely excluding the substantial offline compute required to generate multi-round branching rollouts on development streams (Algorithm 1, lines 2 to 11). Finally, all main results in Table 1 are single-run point estimates without variance across random seeds, and Section 5.2 (line 422) misreferences Figure 3 as Table 3.

Questions for authors
  1. Could you provide the performance and computational cost of an Always-Update baseline that applies the exact same on-policy self-distillation and replay procedure at every decision point? Please also clarify whether Periodic, Random, and Early in Table 1 use the exact same update operator (Equation 13), hyperparameter settings, and replay buffer as LATCH, and explain why Periodic gains only 0.6 points on GSM8K (LLaMA-3.1-8B) despite performing the same number of updates as LATCH.
  2. How is the initial reasoning adapter ϕ0\phi_0 initialized or pre-trained before the online stream begins so that frozen LLaMA-3.1-8B and Qwen3.8-27B can process continuous latent states recurrently? In addition, what is the standalone task performance of the NT=4N_T=4 teacher path compared to the NS=2N_S=2 student path across the online batches?
  3. How are the 16 diagnostic queries partitioned into the fitting and consistency subsets, what is the dimensionality of the orthogonal projection space, and how is the Jacobian GiG_i computed at every decision point without increasing average latency in Table 2? Please also report the computational cost of the development-time branching rollouts used to train the advantage predictor, as well as multi-seed standard deviations on representative benchmarks.
5 comments
AuthorsOpenReview ↗

Response to Reviewer SjS8 (1/3)

We sincerely thank the reviewer for the valuable suggestions and comments. We provide the following point-by-point response.

Q1: Provide the performance and computational cost of an Always-Update baseline; Clarify Periodic, Random, and Early details; Explain the reason for weak gains of Periodic on GSM8K (LLaMA-3.1-8B).

Thank you for the valuable suggestion.

  • Always-Update Baseline. We add an Always-Update baseline that applies the complete OPSD objective in Eq. (13), including on-policy self-distillation and replay, at every nonterminal decision point, while sharing LATCH’s initialization, per-update hyperparameters, and replay protocol and avoiding LATCH’s diagnostic overhead. As shown in Table A, Always-Update outperforms Periodic on MuSiQue, and LATCH achieves higher EM and F1 across both backbones with lower latency, training cost, and total cost, even after accounting for its diagnostic overhead. The results demonstrate that selective evolution provides a more favorable performance–cost trade-off than indiscriminate updating with the same OPSD procedure. These results have also been added into Section 5.2 of the revised manuscript (Table 1 and Table 3).

Table A. Performance and computational cost comparison on MuSiQue across two backbones.

BackboneMethodEM ↑F1 ↑Avg. Latency ↓LLM Calls ↓Train Cost ↓Total Cost ↓
LLaMA-3.1-8BPeriodic23.935.44.841.240.883.25
Always-Update29.440.27.481.692.535.01
LATCH32.543.64.681.780.843.14
Qwen3.8-27BPeriodic41.351.413.491.242.319.06
Always-Update50.259.822.381.807.7215.03
LATCH53.963.113.111.782.308.80
  • Update Protocol Consistency. Periodic, Random, Early, and LATCH use the same update operator (Eq. 13), initialization, teacher construction, on-policy resampling, and optimization settings: eight AdamW steps, effective batch size eight, NS=2N_S=2, NT=4N_T=4, learning rate 10−410^{-4}, τ=1.5\tau=1.5, α=0.5\alpha=0.5, and β=0.3\beta=0.3. They also share the same historical-query sampling and replay-buffer protocols, including buffer initialization, capacity, maintenance, and sampling. All methods follow identical task streams and feedback protocols, differing only in their update schedules. This clarification has also been incorporated into the Implementation Details in Section 5.1 of the revised manuscript.

  • Matched-Count Update Comparison. For the matched-count comparison on GSM8K with LLaMA-3.1-8B, Periodic’s interval is set to match the six updates observed in LATCH’s run. It updates at decision points 3, 6, 9, 12, 15, and 18, without using task outcomes to select individual positions. This controls the update operator and update count while varying the schedule.

    The limited improvement of Periodic highlights the distinction between update frequency and update value. Although Eq. (13) specifies how to incorporate accumulated experience, it does not guarantee that the available corrections are jointly realizable or beneficial for subsequent tasks. Periodic updates at fixed intervals regardless of these conditions, whereas LATCH assesses local realizability, cross-query consistency, and predicted evolution advantage before triggering evolution. Thus, matching the number of updates does not ensure comparable performance gains.

AuthorsOpenReview ↗

Response to Reviewer SjS8 (2/3)

Q2: Clarify the initialization of the reasoning adapter ϕ0\phi_0; Provide the standalone task performance comparison of the NT=4N_T=4 teacher path and the Ns=2N_s=2 student path.

Thank you for the thoughtful suggestion.

  • Reasoning Adapter Initialization. The reasoning adapter is initialized without offline pre-training. As detailed in Appendix A, recurrent latent reasoning is implemented by appending the current latent state as a continuous embedding and extracting the resulting hidden representation from the frozen backbone. The rank-16 reasoning adapter introduces a low-rank residual to this recurrent transition without modifying the backbone’s attention or feed-forward parameters.

    We use zero-output residual initialization, with one low-rank factor randomly initialized and the other set to zero. Thus, the initial adapter contributes no residual, while the frozen backbone performs recurrent latent transitions through the continuous-embedding interface. Subsequent OPSD updates adapt these transitions using accumulated experience, with the backbone remaining frozen throughout. LATCH and all update-timing baselines share the same initial backbone and adapter. These initialization details have also been incorporated into Appendix A of the revised manuscript.

  • Deep Teacher vs. Shallow Student. We evaluate the two-step student and four-step teacher on identical queries using the same adapter checkpoint at each online batch. Both paths independently generate complete answers under identical decoding and scoring settings. Table B shows their task performance aggregated across the online stream on four representative benchmarks and both backbones. The deep path consistently outperforms the shallow path, indicating its role as a stronger reference for on-policy self-distillation. This comparison isolates reasoning depth throughout the online adaptation process. These results have also been incorporated into Section 5.2 of the revised manuscript (Table 5).

Table B: Performance comparison between shallow and deep reasoning paths across online streams on representative benchmarks and different backbones.

BenchmarkBackboneShallow (N_S=2) ↑Deep (N_T=4) ↑Δ\Delta (Deep − Shallow)
GSM8KLLaMA-3.1-8B88.291.7+3.5
GSM8KQwen3.8-27B96.897.6+0.8
HumanEval+LLaMA-3.1-8B69.473.5+4.1
HumanEval+Qwen3.8-27B88.491.3+2.9
BBHLLaMA-3.1-8B67.972.1+4.2
BBHQwen3.8-27B87.990.7+2.8
HotpotQA (F1)LLaMA-3.1-8B52.856.5+3.7
HotpotQA (F1)Qwen3.8-27B71.274.4+3.2
AuthorsOpenReview ↗

Response to Reviewer SjS8 (3/3)

Q3: Clarify the partition of the 16 diagnostic queries and the dimensionality of the orthogonal projection space; Provide the details of the computation of Jacobian GiG_i; Report the the computational cost of the development-time branching rollouts; Provide the multi-seed performance with standard deviations.

Thank you for the valuable comment.

  • Diagnostic Query Partitioning and Projection. At each decision point, we select the 16 most recent queries from the experience history and partition them into eight fitting queries and eight disjoint consistency queries using a fixed-seed random split. The shared parameter change is estimated only on the fitting subset and evaluated on the held-out consistency subset. Latent states are mapped into a fixed 128-dimensional diagnostic space using a Gaussian projection orthogonalized by QR decomposition, distinct from the four-dimensional parameter-update subspace. The projection is generated once and shared across all queries and decision points. These details have also been incorporated into Appendix A and Appendix B of the revised manuscript.

  • Jacobian Computation and Latency. We compute GiG_i using Jacobian–vector products restricted to the four-dimensional update-aligned subspace, requiring four directional derivatives per diagnostic query and avoiding explicit construction of the full parameter Jacobian. These diagnostics are performed only at decision points rather than for every query. Table 3 (Table 2 in the previous version) reports average latency amortized over the entire online stream, including diagnostic reasoning, Jacobian computation, advantage prediction, and triggered OPSD updates. Selective updates help amortize the diagnostic overhead, yielding lower average latency than Periodic under both backbones.

  • Development-Time Computational Cost. We additionally provide the development-time computational cost of training the advantage predictor on MuSiQue in Table C. Training comprises three predictor-training rounds, each with 24 decision points and one paired Update–Wait rollout per point, yielding 144 individual branch rollouts. These costs are incurred before online evaluation and are distinct from the online inference and evolution costs reported in Table 3 (Table 2 in the previous version). The results have also been incorporated into Appendix E.4 of the revised manuscript (Table 6).

Table C: Development-time computational cost of advantage predictor training on MuSiQue.

BackboneBranch Rollouts (#)Rollout Cost (GPU-h) ↓Predictor Training (GPU-h) ↓
LLaMA-3.1-8B14415.30.5
Qwen3.8-27B14442.80.7
  • Multi-Seed Performance. We evaluate all methods over three independent random seeds on four representative benchmarks covering mathematical reasoning, code generation, general reasoning, and multi-hop question answering. Table D shows the mean and standard deviation under LLaMA-3.1-8B. LATCH consistently achieves the highest task performance across all metrics, further supporting the effectiveness and stability of selective evolution across diverse reasoning tasks. The results have also been incorporated into Appendix E.5 of the revised manuscript (Table 7).

Table D: Multi-seed performance comparison on representative benchmarks under LLaMA-3.1-8B.

MethodsGSM8K EM ↑HumanEval+ Pass@1 ↑BBH Acc. ↑HotpotQA EM ↑HotpotQA F1 ↑
Vanilla77.6 ± 0.860.5 ± 1.957.8 ± 1.134.9 ± 1.344.1 ± 1.0
Periodic79.4 ± 1.359.6 ± 2.459.9 ± 1.534.4 ± 1.843.5 ± 1.6
Always81.8 ± 1.465.2 ± 2.062.5 ± 1.538.9 ± 1.748.1 ± 1.8
Random77.8 ± 1.660.3 ± 2.757.7 ± 1.836.5 ± 2.244.7 ± 1.9
Early79.8 ± 1.160.1 ± 1.659.8 ± 1.235.0 ± 1.745.5 ± 1.3
ACE79.6 ± 1.461.2 ± 2.159.5 ± 1.739.9 ± 1.547.6 ± 1.4
REME80.2 ± 1.060.1 ± 1.861.6 ± 1.339.5 ± 2.049.8 ± 1.7
MATTS82.9 ± 1.566.1 ± 2.564.3 ± 1.638.3 ± 2.349.1 ± 2.0
EVOLVER82.3 ± 1.761.9 ± 2.263.5 ± 1.942.6 ± 1.650.2 ± 1.8
LATCH87.6 ± 0.770.3 ± 1.168.5 ± 0.943.6 ± 1.353.5 ± 1.1
Reviewer_BQL7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LATCH, a selective self-evolution framework in the latent reasoning space to determine whether to wait for further experience to accumulate or update the model parameters now using a reasoning adapter. When it favors evolution, LATCH performs on-policy self-distillation to use the deeper reasoning path to supervise the shallow policy with experience replays. The finite-horizon evolution advantage predictor is fitted using the deep-to-shallow corrections along with the latent trajectory encoding. The LATCH framework is evaluated on multiple benchmarks and outperforms various baselines, including update-timing baselines and experience learning baselines.

Critical strengths
  • The latent space is decoded using a reasoning adapter to characterize the agent's experience and can be combined with deep-to-shallow correction diagnostics to determine whether to trigger on-policy self-distillation. The proposed framework answers an important research question during self-evolution: When should the long-horizon agents determine when to evolve themselves, or should they wait for more experience to accumulate?
  • The proposed LATCH framework employs a lightweight predictor to compute the predicted finite-horizon evolution advantage based on the latent reasoning dynamics and is quite efficient in both training and inference compared to the existing baselines.
Critical weaknesses
  • As the entire LATCH framework encodes the trajectories into the latent space, the interpretability of the self-evolution is limited, especially compared to those that experience learning frameworks such as Reasoning Bank, where humans could inspect the learned knowledge from the agent trajectories.
  • My overall impression of the methodology is that it seems over-sophisticated to me, as it combines multiple components in the proposed framework. It instantiates the agent parameters as a backbone and a trainable reasoning adapter; it uses deep-to-shallow diagnostics to estimate the finite-horizon advantage prediction and employs on-policy self-distillation. There are multiple hyperparameters in the framework, and as we can see, the performance is sensitive to those parameters.
Questions for authors
  • The proposed LATCH framework is mainly evaluated on math reasoning, code generation, and multi-hop reasoning tasks. I'm wondering whether the authors have evaluated it on more agentic tasks, such as WebArena and SWE-Bench, where experience learning is more important for solving the tasks correctly and efficiently?
7 comments
AuthorsOpenReview ↗

Response to Reviewer BQL7 (1/2)

We sincerely thank the reviewer for the valuable suggestions and comments. We provide the following point-by-point response.

Q: Evaluate LATCH on more agentic tasks, such as WebArena and SWE-Bench, where experience learning is more important for solving the tasks correctly and efficiently.

Thank you for the constructive suggestion.

To evaluate LATCH beyond conventional reasoning benchmarks, we additionally conduct experiments on SWE-bench Verified and WebArena, covering repository-level software engineering and multi-step web interaction, as shown in Table A. We evaluate LLaMA-3.1-8B and Qwen3.8-27B against all baselines, with results averaged over three independent runs. The evaluation follows sequential online task streams, allowing experience-learning methods to leverage accumulated experience across tasks.

Table A: Task performance on long-horizon interactive agent benchmarks across different backbones. SWE-bench Verified is evaluated by Resolved Rate (%), and WebArena by Success Rate (%).

MethodsSWE-bench Verified (LLaMA-3.1-8B)SWE-bench Verified (Qwen3.8-27B)WebArena (LLaMA-3.1-8B)WebArena (Qwen3.8-27B)
Vanilla10.925.112.427.8
Periodic12.227.314.129.5
Random11.526.413.928.2
Early12.627.914.530.4
ACE16.732.818.634.5
ReMe18.434.220.536.2
MaTTS19.836.822.839.8
EvolveR19.235.921.638.4
LATCH23.540.526.742.6

LATCH consistently achieves the best performance across benchmarks and backbones, demonstrating its effectiveness in long-horizon interactive environments, where agents coordinate sequential actions, incorporate environmental feedback, and leverage accumulated experience to improve subsequent decisions. The performance results have also been incorporated into Section 5.2 of the revised manuscript (Table 2).

AuthorsOpenReview ↗

Response to Reviewer BQL7 (2/2)

For computational efficiency, we further evaluate inference and evolution costs under LLaMA-3.1-8B, as reported in Table B, using the same metrics as Table 3 of the manuscript (Table 2 in the previous version).

Table B: Efficiency and computational cost on interactive agent benchmarks.

BenchmarkMethodsAvg. Latency ↓LLM Calls ↓Train Cost ↓Total Cost ↓
SWE-bench VerifiedVanilla312.523.6—45.2
Periodic413.626.19.660.5
Random421.226.510.261.9
Early383.725.88.257.6
ACE765.143.229.4119.7
ReMe364.727.44.554.1
MaTTS1563.498.711.8207.6
EvolveR2208.986.3121.7307.8
LATCH391.530.28.159.3
WebArenaVanilla142.413.8—26.7
Periodic194.915.95.236.6
Random207.316.45.637.9
Early195.715.54.635.4
ACE388.526.816.570.1
ReMe177.316.82.332.0
MaTTS982.748.96.9104.3
EvolveR1316.962.570.3192.7
LATCH193.619.64.536.2

LATCH achieves higher task success than computationally intensive baselines such as MaTTS/EvolveR while incurring lower inference and evolution costs. Its overall cost remains comparable to lightweight update schedules, indicating that selective evolution enables effective experience adaptation without excessive computational overhead. The efficiency results have also been incorporated into Section 5.2 of the revised manuscript (Table 4).

These results show that LATCH generalizes to long-horizon interactive agent tasks, consistently improving task performance across different backbones while maintaining a favorable trade-off between adaptation effectiveness and computational cost.

AuthorsOpenReview ↗

Looking forward to your feedback

Dear Reviewer BQL7:

Thanks for your valuable comments. We would be grateful if you could allocate some time to review our responses.

We understand that you have a multitude of responsibilities. To facilitate a swift evaluation of our responses, we have summarized the primary changes as follows,

  • We have added performance evaluations on two long-horizon interactive agent benchmarks, SWE-bench Verified and WebArena (Table A).

  • We have also added computational efficiency analyses on both interactive benchmarks under LLaMA-3.1-8B (Table B).

These additional experiments and analyses have been incorporated into our revised manuscript.

We would like to express our heartfelt gratitude for the time and effort you have dedicated to reviewing our work. We are eager to ensure that we have adequately addressed your concerns. Please let us know if you have any additional concerns or questions. We would be glad to discuss them.

Should you find that our responses have satisfactorily addressed your main concerns, we kindly request that you re-evaluate our paper based on the provided responses.

Thank you again for your time and consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.