acceptodds
Under review as a conference paper at ICLR 2027

Runtime Learning in Open World: Adaptive Blending and Cognitive Sampling

Abstract

This paper proposes OpenRun, a framework that enables safety-critical autonomous systems to perform runtime learning in open-world environments. Here, open-world environments refer to environments whose states, conditions, and operational scenarios are not fully known prior to deployment and may continuously evolve or emerge during operation. OpenRun comprises a Learning-Student, a Safety-Teacher, and an Action-Blender. The Learning-Student builds on actor-critic reinforcement learning and introduces a novel cognitive batch sampling strategy that prioritizes experiences based on temporal recency, occurrence frequency, and contextual similarity, significantly enhancing learning efficiency. The Safety-Teacher operates in parallel with the Learning-Student, providing verifiable safety protocols that enable safe runtime learning across a broad range of action policies, from purely control to vision-to-action policies. The Action-Blender introduces a novel safety- and exploration-aware blending mechanism that adaptively blends the actions of the Learning-Student and Safety-Teacher to enhance safety assurance while preserving exploration. We further establish theoretical foundations demonstrating how these innovations collectively enable safer and more efficient runtime learning in open-world environments. Extensive experiments with a quadruped robot and an off-road autonomous vehicle in challenging real-world and simulated wild-forest environments validate the effectiveness and distinctive capabilities of OpenRun, as well as its generalizability across functions, platforms, and environments.

Primary area:
reinforcement learning
Keywords:
Runtime Learning, Actor-Critic Reinforcement Learning, Action Blending, Experience Sampling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_XdQ9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OpenRun, a framework for runtime learning in safety-critical open-world environments. OpenRun consists of a Learning-Student with cognitive batch sampling, a Safety- Teacher for safe planning and control, and an Action-Blender that adaptively combines their actions to balance safety and exploration. Experiments on a quadruped robot in real forests and an off-road vehicle in simulation demonstrate improved learning and task performance.

Critical strengths

S1.The paper presents a well-motivated framework that jointly addresses safety and learning efficiency in open-world environments, with theoretical analyses and ablation studies supporting the proposed adaptive blending and cognitive batch sampling mechanisms. S2.The evaluation on a physical quadruped robot in challenging real-world forest environments, together with comparisons against multiple safe RL baselines, provides valuable evidence of the framework's practical effectiveness. S3.The paper is well-organized and clearly written. The motivation is clear. Figures and tables adequately illustrate the proposed framework and experimental results. W1.

Critical weaknesses

W1.The paper lacks sensitivity analyses for several key parameters, including the blending parameter α in Eq. (10) and the cognitive sampling parameters C, c, λ, and ξ in Appendix H.1.3. The authors should evaluate how these parameters affect safety, exploration, and learning performance to demonstrate the robustness of the proposed mechanisms. W2.Appendix H.4 reports the execution frequencies of different modules but does not clearly specify how these frequencies correspond to individual operations such as experience sampling, policy inference, and gradient updates. The paper also lacks quantitative measurements of sampling latency and computational overhead. The authors should clarify the execution schedule and report the corresponding runtime costs. W3.Figure 5 evaluates frequency and recency only as a combined factor (F-R), without examining their individual contributions. The current ablation study does not establish whether the improvements arise from frequency, recency, or their combination. The authors should include separate frequency-only and recencyonly variants to provide a more complete analysis.

Questions for authors

W1-W3

Reviewer_Ky4bOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper addresses an important problem: making policies more adaptive in open-world environments. It proposes an integrated framework comprising a safety teacher, a learning student, and an action blender.

Critical strengths
  • Safe adaptation after deployment is an important problem with substantial practical relevance and potential for follow-up research.

  • Forest navigation provides encouraging evidence of feasibility, with higher task completion than the evaluated baselines.

Critical weaknesses
  • The contribution appears primarily integrative, with relatively incremental advances in individual components. A clear, unified insight that others could readily build upon is missing.
  • As a systems contribution, the paper needs broader and more thorough real-world evaluation to demonstrate its value under realistic deployment conditions.
  • The existing baseline comparisons do not sufficiently isolate where the gains come from or establish the benefits of the proposed integration over simpler alternatives. Teacher-only and frozen-policy baselines are needed to isolate the benefit of runtime learning, while Residual-Cognitive would complete the blending/replay ablation. The replay analysis assumes conditionally uncorrelated gradient noise despite selecting similar, recent transitions; neither this assumption nor reduced gradient mismatch is empirically checked.
  • Cross-environment transfer lacks quantitative comparisons against frozen or reset policies, including adaptation speed and retention. The 15-trip real-world evaluations also need clearer reset/matching protocols, uncertainty on success rates. Wall-clock adaptation time and measured control latency would better support the runtime-efficiency claims.
Questions for authors
  • How broadly can this framework be applied to real-world robotics? Additional experiments across diverse robotic tasks would help clarify its practical applicability and limitations.
Reviewer_w4zyOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors propose the OpenRun framework for runtime learning of safety-critical autonomous systems in open-world environments. It consists of 1) an Action-Blender that mixes the actions of the two other modules with a state-dependent weight, 2) a Learning-Student, an actor-critic agent that replays experiences selected by a cognition-inspired ranking of similarity, frequency, and recency, and 3) a Safety-Teacher based on a planner and an adaptive controller. They evaluate OpenRun on a real quadruped robot in wild forests and on an off-road vehicle in simulation.

Critical strengths

The authors evaluate runtime learning on a real quadruped robot in wild forests, running end-to-end on an onboard edge device, and provide field-experiment videos. Experiments in such unstructured real environments are rare in this line of work.

Critical weaknesses
  1. My main concern is that Section 4.3 does not support the claimed benefit of cognitive batch sampling, which is presented as a main novelty. Theorem 2 relies on strong assumptions, and more importantly, the bound does not convey meaningful information. The contextual mismatch term does not vanish even when every selected experience coincides with the current transition, so the bound cannot be small. With cosine similarity, which the authors list as a candidate, SS can be negative, and it is not explained how this is handled in Eq. (11) or in Assumption 2.

  2. Since frequency and recency are never defined mathematically, the interpretation in Remark 3 cannot be justified from Theorem 2. Corollary 2 also describes an overly special situation. Its sufficient condition requires the added experience to be no more recent than the most recent existing one, which excludes a genuinely new experience. For Nj=1N_j=1, the condition already seems to presuppose a special situation, and the authors should explain this part in detail.

  3. In addition, the design of cognitive batch sampling is not explained in sufficient detail, and the notation is not clarified. The experience instance in Eq. (14) appears without motivation, and neither the background of the memory decay factor λ\lambda nor the reason why the frequency and recency measures satisfy the relation in Eq. (17) is given. The symbol RR is used both for the reward in Eq. (12) and for the recency measure in Eq. (11).

Overall, the current manuscript gives the impression of having been written in haste, and many sentences are not reader-friendly, which makes it difficult to grasp their intended meaning. The authors should revise it thoroughly so that each quantity is defined and motivated before it is used.

Minor issues

  • Section 4.2: the term "measure functions" is misleading, since these are just scoring functions rather than measures.
  • Appendix F.2: the batch is written as BB in K[j]BK^{B}_{[j]} but as B\mathcal{B} elsewhere.
  • Appendix I.2 refers to a "Learning-Agent or Rational-Agent", which are not defined in the paper.
  • Section 6.1: the links to the videos do not open.
Questions for authors
  1. Are the images at the bottom of Figure 1 rendered from the actual simulator environments?
  2. Assumption 1 requires the dynamics to be affine in the action. How restrictive is this for the quadruped robot and the off-road vehicle in the experiments?
  3. Under what conditions is the model-mismatch variation small enough for Eq. (25) to hold, and how is α^\hat{\alpha} defined?
2 comments

Thank you for your sincere and detailed responses, and for the substantial effort you put into revising Section 4. Your responses have resolved my questions and have partially addressed my concerns about the weaknesses. Before going further, I would like to check: In your response to W1, you state that "Recency-Frequency-Only Sampling addresses only contextual error", and I wonder whether you meant that it addresses only the temporal error.

However, I am still not fully convinced regarding W1 and W4, and I think the main reason is that the definitions of the notation remain unclear.

  • In Eq. (11), it is not easy to see why the scoring functions for similarity, frequency, and recency can be combined additively,
  • In Eq. (14) and Eq. (17), it is hard to tell at a glance whether the design choices are derived from theory or are heuristic.

I suspect that this difficulty comes from the structure of the paper as a whole. Because the paper presents three modules together with their theoretical analyses at the same time, the manuscript feels fragmented. Although theory-driven design is valuable, placing all analyses alongside the method obscures the main message. If you move the theoretical analyses to the appendix and instead use the main text to explain the design rationale of the three modules and how they work together, I believe the paper would be much easier for readers to follow.

In addition, I would encourage you to highlight your real-world experiments more in the main text, as I consider the most notable element of your paper. At present they appear only as links, which makes this contribution look minor. Including highlights from the videos in the experiment section, together with a qualitative analysis alongside the quantitative results, would make the distinctive contribution of your work much more visible. At this stage, I will maintain my initial rating.

Q1: In your response to W1, you state that "Recency-Frequency-Only Sampling addresses only contextual error", and I wonder whether you meant that it addresses only temporal error.

Response: It was a writing error. It should be temporal error.


Q2: In Eq. (11), it is not easy to see why the scoring functions for similarity, frequency, and recency can be combined additively.

Response: The additive combination in Eq. (11) is a design choice for constructing a unified experience-ranking score, rather than an assumption that similarity, frequency, and recency are statistically independent or have additive effects on learning performance. Combining these complementary signals additively provides a simple, interpretable way to balance their contributions when selecting experiences. Combining them is grounded in cognitive foundations that reveal humans' fast and efficient learning:

  • Frequency: The human mind naturally prioritizes familiar and frequently encountered information, facilitating memory retrieval and reducing cognitive effort, thereby enhancing cognitive efficiency.
  • Recency: Human memory exhibits a bias toward recently accessed information, enabling faster retrieval and more effective decision-making in complex and non-stationary environments.
  • Similarity: The human brain uses surface-level and structural cues to retrieve memories, with structural similarity dominating relational knowledge retrieval and supporting efficient generalization.

Theoretically and experimentally, our studies indicate that there may exist an integrated science of mind: Human learning combines them, while runtime learning for machine intelligence achieves significant performance gains by jointly considering them when sampling memories for learning.


Q3: In Eq. (14) and Eq. (17), it is hard to tell at a glance whether the design choices are derived from theory or are heuristic.

Response: They are neither derived from theory nor purely heuristic. Specifically, as cited in the paper, Eq. (17) is a variant of the temporal dynamics model of human declarative memory that simultaneously captures the effects of frequency and recency on memory retrieval, calibrated for runtime learning. The Eq. (14) is only to map high-dimensional experience instances to scalar scores for ranking.


Q4: If you move the theoretical analyses to the appendix and instead use the main text to explain the design rationale of the three modules and how they work together, I believe the paper would be much easier for readers to follow.

Response: We appreciate the reviewer's suggestion. Our initial version was trying to follow the structure. However, reviewing the results in the appendices is optional for reviewers. It can also impress reviewers that the paper has no theoretical contributions.


Q5: At present they appear only as links, which makes this contribution look minor. Including highlights from the videos in the experiment section, together with a qualitative analysis alongside the quantitative results, would make the distinctive contribution of your work much more visible.

Response: Thank you for this constructive suggestion. We agree that presenting the videos only as links does not adequately highlight the distinctive contributions of our work. The videos provide proof of the robot's travel time, paths, .... Due to the page limit, some representative frames are provided in the paper; for example, Figure 9 highlights learning gains.

Reviewer_B3x7OpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The goal of this paper is to enable safe and efficient runtime learning in open-world environments. The approach is a blending of teaching and learning that allows safe learning in physical situations. The paper's experiments involve a quadraped robot that engages with runtime learning in open world environments. The video demonstrations show the behavior of the experiments that exemplify the results of the experiments. Some theoretical exploration is provided for the effectiveness of the learning approaches.

Critical strengths
  • The paper takes a unique approch to allow several active modules to collaborate in open-world environments that build upon preliminary training. The Safety-Teacher is designed based on first principles, rather than learning-informed, to enable the runtime learning system to operate in various policies. The Learning-Student can operate within the envelope of the safety-teacher to utilize its previous experiences and ongoing experiences. The Action-Blender can impact the learning-student's ability to execute within the safety envelope, to avoid the learning-student from simply "bumping up" against the safety margin and therefore not exploring the design space during execution.
  • Among the contributions of the paper, the Action-Blender is the most interesting. When applying an RL controller in an unsafe system for testing, a common reaction is for the RL controller to learn the safety bounds: in simulation this can be effective. However, in transition between the simulation environment to a physical environment, it may be that the safety controller will continuously be executing the system, and filtering all the commands from the RL controller, thus nullifying any impact of the safety system.
Critical weaknesses
  • I found the context and justification for the learning-student module a bit more challenging to justify. The reliance on the frequency and recency may reflect human learning, but may be seen as physiological limitations rather than design goals. If the goal of the system is to replicate human learning, they could be useful: however, the limitations of human memory may be mitigated with additional computing power. Without additional justification for this approach, the goal of including this as something like a first-class design principle is secondary to the benefits of the other pieces of the design.
  • The paper could go further in explaining: why is the contribution including the learning-student as well as the safety-teacher. I see the principal contribution of the work is the action blender---and I think that with the approach taken in the paper, this could be useful in many other other learning schemes (even beyond RL), and can operate with many different safety-teacher. The paper could be strengthened in explaining more about why all three pieces are needed to justify the work.
Questions for authors

Why are all three pieces are needed to justify the work? There could be a misunderstanding from my perspective, but I see that many different learning-student and safety-teacher could be involved, but still benefit from the action-blender. What is the unifying reason why all three need to be designed as such, vs. applying to other approaches?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.