acceptodds
Under review as a conference paper at ICLR 2027

Bi-Level Adaptive Regulation of Intra- and Inter-Task Learning for Compact Long-Horizon Agents

Abstract

Developing agents that autonomously complete long-horizon tasks is an important goal in AI research. Repeated calls to large-scale language models during long-horizon interactions can incur high inference costs, motivating the development of capable agents through targeted training of compact language models. However, existing training paradigms provide limited support for adaptive regulation within and across tasks. At the intra-task level, outcome rewards or undifferentiated process rewards offer limited guidance for switching among reasoning modes. At the inter-task level, fixed task distributions can become misaligned with agents' evolving capabilities. As a result, compact agents may persist with unsuitable reasoning modes and repeat ineffective actions, increasing interaction costs without effectively advancing the task. To this end, this paper proposes ARETE (Adaptive REgulation with Tailored Experiences), a long-horizon reinforcement learning framework that combines intra-task reasoning regulation with inter-task practice regulation. For intra-task regulation, inspired by the self-regulated learning (SRL) framework from educational psychology, ARETE organizes reasoning into three modes: forethought, performance, and self-reflection, using differentiated, verifiable feedback to regulate their use. For inter-task regulation, inspired by the zone of proximal development from developmental and educational psychology, ARETE generates verified scaffolded tasks and compares unaided and scaffolded performance to estimate learning potential, guiding capability-matched task generation and practice allocation. Experiments on two complex interactive benchmarks show that ARETE improves compact agents, enabling a 2B-parameter model to perform competitively with models exceeding 100B parameters.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Long-Horizon Agents, Adaptive Reasoning, Adaptive Regulation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_m3UPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

ARETE trains compact language model agents through two forms of regulation: choosing between planning, executing, and reflecting on tasks, and adjusting practice tasks by utilizing the difference between independent and scaffolded performance. According to experimental reports from TextCraft and ScienceWorld, the success rate is higher and there are fewer ineffective operations compared to the selected baseline.

Critical strengths

1.Separating whether the reasoning mode is appropriate from whether it is executed correctly can address a specific weakness in outcome-only training.

2.The ablation of components also indicates that both reasoning regulation and adaptive practice contribute, with the former generating greater benefits.

Critical weaknesses

1.Lack of the closest baseline. RLVMR has combined structured reasoning behavior with verifiable process rewards. Although discussed in the paper, it was not evaluated, resulting in the benefits of the ARETE additional mechanism remaining unresolved.

2.The replicated GRPO baseline is worth carefully examining: Qwen2.5-3B only achieved 64% on TextCraft, while the proportions in AgentGym-RL and HPO papers were 75% and 81%, respectively. These settings may not be exactly the same, but the unexplained differences weaken the claimed improvement.

3.Matching the number of policy rollouts does not establish equal training resources. ARETE also uses teacher assisted demonstrations, scaffolded tasks, task generation, and simulator validation; These comparisons do not clearly separate algorithm benefits from these additional resources.

4.The broader generalization evidence is still limited to two environments. The provided text neither reports the differences between training seeds nor provides uncertainty estimates, making it difficult to assess the reliability of the benefits.

Questions for authors

1.How does ARETE compare to RLVMR under the same backbone, initialization, task split, and training budget?

2.What is the reason for the lower TextCraft GRPO results compared to previous reports? Has the original implementation and training plan been replicated?

3.Will the benefits continue when the baseline receives equal demonstration and scaffolded training data? Please also report the total training cost and results for multiple seeds.

Reviewer_6yPuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ARETE, a bi-level reinforcement learning framework with intra-task self-regulatory reasoning control and inter-task adaptive training practice. Experiments across two complex multi-turn interactive benchmarks show consistent task-success gains across two compact backbones, enabling a 2B model to perform competitively with larger models.

Critical strengths
  1. The proposed method is well-motivated. It decouples supervision for reasoning mode selection and execution quality, and leverages scaffolded tasks to implement adaptive curriculum sampling based on learnable potential.
  2. Empirical results demonstrate that the trained compact agents surpass same-scale baselines and achieve competitive performance against large open-source models
Critical weaknesses
  1. Many details are insufficiently specified. How are the mode-applicability feedback and execution-quality feedback computed in practice? How are the associated weights and calibration parameters determined? Moreover, how does the agent decide which reasoning mode to select at each interaction step? More detailed descriptions are needed to improve the clarity of the method.
  2. The inter-task regulation mechanism relies on scaffolded task conditions to estimate learning potential and guide task allocation. However, the paper does not sufficiently explain how the additional goal-decomposition guidance is obtained, whether it requires an external teacher model or manual effort. Moreover, the ablation study does not isolate the contribution of the proposed learning-potential score from other components of inter-task regulation.
Questions for authors
  1. Could the authors provide more detailed ablation studies on the proposed learning-potential score?
  2. How is the goal-decomposition guidance obtained, and does it consistently improve task performance? Is the performance gap a reliable indicator of learning potential?
Reviewer_ES1KOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ARETE is an RL framework for training compact long-horizon LLM agents, combining reasoning-mode supervision with adaptive task sampling (curriculum). Experiments on the toy environments TextCraft and ScienceWorld show gains over several baselines.

Critical strengths

The paper addresses the relevant problem of training compact LLM agents for multi-step tasks.

Critical weaknesses

W1: The baseline comparison is questionable: ARETE receives an SFT cold start, richer behavioral instructions, and scaffolded/generated practice, whereas the Qwen baselines start from the original checkpoint. I recommend to enhance evaluation with SFT -> GRPO baseline with a simple curriculum (e.g., based on PLR [1]), using the same demonstrations, equivalent behavioral instructions, and matched data access. Otherwise, the gains cannot be clearly attributed to proposed techniques.

W2: Evaluation is limited to two toy text-based environments, TextCraft and ScienceWorld, with a small interaction limit. This provides limited evidence for the broader claims about long-horizon agents. Evaluation on more realistic tasks and longer horizons will be helpful.

W3: Appendix B reports a fixed seed of 11, and no multi-seed results or uncertainty estimates are provided. This makes it difficult to assess the robustness of the gains.

W4: The paper does not report actual training and inference token usage for each approach. Matching rollout counts does not match computational budgets, as appendix acknowledges, where ARETE could produce longer reasoning traces.

W5: The paper does not establish how much manual engineering is required to transfer proposed approach to new domains, given its reliance on mode-specific verifiers, goal-consistency checks, and reference-action replay.

W6: The need for the SRL cold start is unclear. On Qwen2.5-3B, removing it reduces success by only 2 percentage points on TextCraft and 2.5 on ScienceWorld (Table 3). The paper should justify whether this benefit warrants teacher-assisted demonstrations and an additional training stage, or show benefits in training stability or sample efficiency. Its inclusion also complicates comparisons with baselines trained without that stage.

W7: The framework diagrams are overcrowded, with too many elements and very small text, making them difficult to read at normal page size. Simplifying the diagrams and enlarging the labels would make the method easier to follow.

W8: ScalingInter-RL provides a fixed interaction-horizon curriculum (10 -> 20 -> 30 turns), but the experiments omit adaptive task-selection baselines such as PLR [1] and curriculum methods for reinforcement learning fine-tuning, including AdaRFT [2], VCRL [3], and ADCL [4].

[1] Jiang M, Grefenstette E, Rocktäschel T. Prioritized level replay. InInternational Conference on Machine Learning 2021 Jul 1 (pp. 4940-4950). PMLR.

[2] Shi T, Wu Y, Song L, Zhou T, Zhao J. Efficient Reinforcement Finetuning via Adaptive Curriculum Learning. Transactions on Machine Learning Research. 2026.

[3] Jiang G, Feng W, Quan G, Hao C, Zhang Y, Liu G, Wang H. Vcrl: Variance-based curriculum reinforcement learning for large language models. arXiv preprint arXiv:2509.19803. 2025 Sep 24.

[4] Zhang E, Yan X, Lin W, Zhang T, Qianchun L. Learning like humans: Advancing llm reasoning capabilities via adaptive difficulty curriculum learning and expert-guided self-reformulation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025 Nov.

Questions for authors

Q1: Why does ARETE approach start substantially higher in Figure 6, compared to baselines?

Q2: Can you provide the missing SRL-SFT demonstrations and training recipe, plus configurations for the baselines, ScienceWorld, and Qwen2.5-3B experiments?

Q3: How were the SFT and RL training budgets selected? Were they fixed in advance or chosen using validation performance, and do the baselines converge within these budgets?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.