acceptodds
Under review as a conference paper at ICLR 2027

Real-Time Heuristic Learning: Compiling Online Evidence into Executable Policies

Abstract

Real-time agents must act promptly, but handling changing states does not establish adaptation to changing rules. We introduce Evolving Freeway and Evolving Snake, extending Real-Time Reasoning Gym with repeated hidden rule changes within continuing episodes. Phase boundaries are public and active rules hidden. Transitions provide evidence, and world state persists. Reactive, Planning, and AgileThinker retain discoveries as prompt evidence, action sequences, or textual guidance rather than installed policy revisions. We propose Real-Time Heuristic Learning (RTHL), an evidence-to-code framework compiling transition evidence into reusable executable rules. A deterministic heuristic selects actions. Post-transition diagnosis drives bounded code revisions, screened by program checks and replay, with phase scoping and rollback governing reuse. Across 576 episodes, RTHL leads all six pooled comparisons of mean terminal scores across environments and budgets, gaining 8.5–12.7% over Reactive on Freeway and 43.4–49.4% on Snake. At 1K, updating the same fixed initial controller adds 5.00 Freeway points and 10.25 Snake points. RTHL leads 12 of 14 Hard comparisons across seven budgets (1K–32K), including seven on Snake. RTHL enables reusable adaptation without weight updates.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language model agents, real-time decision making, online adaptation, executable policies, program synthesis, non-stationary environments

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fMquOpenReview ↗
Rating
1
Confidence
3
Informativeness
2(+1)
Presentation
1
Soundness
2
Read review
Paper summary

This work focuses on the interesting topic of Heuristic Learning, proposing one benchmark and one scaffold. The benchmark extends Real-Time Reasoning Gym from changing states to changing rules, with two environments: Evolving Freeway and Evolving Snake. Each episode has three 30-action phases. The phase boundaries are public, the active mechanism is hidden, and world state persists across boundaries.

The proposed method, RTHL, uses code as an evolving harness:

  • A deterministic controller selects every action.
  • An LM Critic diagnoses evidence after each transition.
  • A coding agent then installs validated, phase-scoped revisions to that controller.

The experiments cover:

  • a main comparison across both games, three difficulty levels, and several token budgets;
  • a matched ablation against the same controller with online updates disabled;
  • component ablations of replay/rollback and phase scoping;
  • an extended budget sweep on the Hard setting;
  • token-usage accounting.

RTHL outperforms three reasoning baselines (Reactive, Planning, and AgileThinker), with the clearest advantage on Snake. It also improves over its own fixed starting controller, and it generates less LM output than the baselines. A within-phase reward-shift analysis and installation-timing traces show adaptation happening within a regime.

Critical strengths
  1. A well-motivated problem and a natural design answer to it.
  • The paper identifies a real gap in existing real-time agent benchmarks. They test whether an agent can react to changing states, but not whether it can discover and reuse changing rules within one continuing episode (§1; App. D, Table 20).
  • The two Evolving environments target exactly this gap. Rules switch while the world state persists, and the agent must infer the switch from outcomes alone. This is a useful extension of Real-Time Reasoning Gym (§2.1, Fig. 1, Table 1; App. A).
  • The scaffold separates acting (an installed deterministic controller) from revising the policy (Critic diagnosis followed by validated code edits; §3, Fig. 3). Existing approaches (Reactive, Planning, AgileThinker) carry what they discover forward only as prompt text, queued actions, or textual guidance (§3.1, Table 2). RTHL instead turns a discovery into code that is re-applied to every new state and switched off when its phase ends (§3.2–3.4; Eqs. 1–5).
  • This removes the LM from the per-step action path. That is a sensible response to the latency-versus-quality tension that motivates real-time reasoning benchmarks, and it also reduces how much LM output is generated per episode compared with the baselines (§4.3, Table 4; App. C.3).
  1. Controlled experiments that attribute the gain to online revision and locate it in time.
  • A matched ablation keeps exactly the same starting controller and memory and only disables the LM revision loop. Full RTHL beats this fixed version in both games and in every difficulty setting (§4.2, Table 3; App. C.4, Table 17).
  • The fixed controller on its own does not beat the strongest baseline. So the advantage over existing methods comes from the online updates, not simply from the hand-written controller (App. C.4, Table 17 vs. App. C.1, Tables 6–7).
  • The advantage over the three baselines is consistent on Snake across difficulty levels and token budgets. It holds even at large budgets where the Planning baseline finally produces usable plans, and it grows as the navigation constraints get harder (§4.1, Fig. 4; App. C.1, Table 7; §4.3, Fig. 5; App. C.2, Tables 8 and 10).
  • The paper goes beyond final scores:
    • A within-phase analysis compares reward at the start and end of each rule phase against the fixed controller. It shows that performance improves within a regime, not merely across episodes (§4.2; App. C.5, Eq. 6, Table 18).
    • Installation-timing traces show that most phases receive a usable code revision early enough to be reused before the rules change again (App. C.5).
    • Logged execution cases trace a single diagnosis through validation, installation, reuse on later states, and deactivation at the phase boundary. This makes the mechanism concrete and auditable (App. E, Tables 21–22).
Critical weaknesses
  1. The writing and organization make the paper hard to understand, especially for readers outside this sub-area. This is my main concern, and it affects how far the results can be checked at all. (The full itemized list is in the Presentation field.)
  • Key concepts are undefined or overloaded.
    • "Hazard" is never defined, yet it is used for four different things: the ordinary word, the HAZARD benchmark, one of the paper's hidden rules, and a code hook. The paper also lists obstacles and hazards side by side in §2.1 without explaining the difference. A reader only learns that hazards are hidden lethal cells from an execution case on p.24.
    • "Deployment-time updating" (a stated contribution, l.121) is never defined.
    • The abbreviation E/M/H is used without being introduced.
  • The task and the baselines are under-specified.
    • The hidden rules are described in a few words each, for example "make positive-food columns dangerous". From this, the reader cannot tell what an agent must actually learn, or how hard each rule is.
    • Reactive, Planning, and AgileThinker each get one or two compressed sentences (§3.1, Table 2). As a result, it is hard to interpret their results, for example why Planning scores strongly negative on Snake.
    • Fig. 1 does not even state the objective, which is to maximize crossings or net food reward within a fixed horizon.
  • Information the reader needs is scattered. Facts required to read the headline results sit in appendix paragraphs or are missing entirely:
    • that RTHL's LM calls are not charged to the real-time clock (App. A, "Temporal Execution");
    • that all agents are given a catalog of rule families (App. A);
    • how Fig. 2 is normalized (not stated). Meanwhile, §3.3–3.4 introduce a dense set of symbols that are mostly used once, and Fig. 4 spends most of a page re-plotting Tables 6–7.
  • The presentation is not well organized or polished.
    • Table 2 entries break mid-word.
    • Fig. 3 mixes text labels and decision diamonds for its branches.
    • Citations are not parenthesized, and there are many spacing errors.
    • The prose relies heavily on unexplained jargon ("architecture-native", "logical release", "action authority").
  • Taken together, the paper reads as written for readers who already know the codebase, more likely a tech note. The idea itself is not complicated, but a reader unfamiliar with Real-Time Reasoning Gym or this domain cannot reconstruct the task, the baselines, or the evaluation protocol from the main text.
  1. The evaluation, as designed, does not support the "real-time" and "learning" claims.
  • The real-time constraint is not applied to RTHL's adaptation.
    • In the baselines, the token budget B determines which reasoning is available at each decision point.
    • For RTHL, B only caps each Critic or coding call, and continuation is allowed. Its adaptation "completes before the next environment step" (l.151–152), and its LM calls "block advancement of the Python loop" (App. A).
    • So RTHL diagnoses and rewrites code while the world is effectively paused. This is exactly the pause-during-deliberation setting the paper criticizes in WebArena/KORGym (l.454).
    • A symptom: Snake-H RTHL scores are identical at six of seven budgets (Table 8).
    • A fair protocol would charge each call's generated tokens to logical time, with the old controller acting in the meantime.
  • The "learning" is hard to separate from the engineering.
    • RTHL starts from a hand-written, environment-specific search controller (App. B).
    • Edits are restricted to named hooks that map almost one-to-one onto the implemented rule families.
    • Each hook has hand-tuned validators.
    • The catalog of rule families is supplied to the agent (App. A).
    • The task is therefore closer to identifying which known mechanism is active and filling a pre-built slot than to compiling open-ended evidence into code.
    • The most informative baseline is missing: the same controller plus a simple non-LM consistency test over the catalog.
  • The evidence base is narrow.
    • Every LM role uses a single model, DeepSeek-V4-Flash.
    • The Freeway advantage over Reactive is within noise.
    • The replay/rollback and phase-scoping components show no measurable effect (Table 17).
Questions for authors

Q1 (evaluation protocol and time accounting). The main text does not explain how RTHL's LM calls relate to the real-time clock; this has to be pieced together from App. A. Please state explicitly:

  • how many environment steps elapse while a Critic call plus a coding call is generated;
  • whether this time is ever charged to RTHL. Could you also report results under a protocol in which each call's generated tokens consume logical steps at the same B used for the baselines, with the old controller acting in the meantime?

Q2 (exact rule semantics). Please add a per-family table to the paper giving exact definitions and parameters. For the Snake column hazard in particular:

  • Is the whole column of each positive food lethal?
  • Does the danger follow the food as it moves, expires, or is eaten?
  • Are several columns lethal at once?
  • Is the food cell itself lethal? For Freeway, please give the lane-stay threshold, the collision-width increase, and the reversal and two-pass periods. Without these, a reader cannot judge what the agent has to learn.

Q3 (baselines). Please describe concretely how Reactive, Planning, and AgileThinker work in your setting:

  • the prompt and output format; the release mechanism under B; the default action in each game; how AgileThinker's fast branch uses the slow trace; how the three differ from each other.
  • Relatedly, is the default action itself lethal in Snake? This would explain Planning's strongly negative scores (Tables 7, 19).
  • Evaluate with additional backbone LMs. A cheap but strong model (e.g., GPT-6-Luna) and a small open model (e.g., Qwen3.8-27B) would test whether RTHL degrades more gracefully than Reactive as the LM becomes weaker.

Minor questions:

  • Budget invariance: Why are the Snake-H RTHL scores identical at six of seven budgets (Table 8), while Freeway-H varies non-monotonically (Table 9)? What sampling temperature does each role use?
  • Freeway significance: Please report paired CIs for RTHL vs Reactive in each cell (Table 6), as you do for Full vs Fixed.
  • Identification accuracy: From your logs, can you report how often the installed hook matches the true active rule, the false-install rate, and a per-family breakdown?
  • Non-LM baseline: How would the same controller with a hand-coded consistency test over the known rule families perform?
  • Seed selection: How were the non-contiguous Freeway seeds chosen? Were the controllers, hooks, and validators developed on these same seeds?
  • Guarded consolidation: Replay/rollback and phase scoping show no score effect (Table 17). Do they reduce harmful installs or variance?
  • Fig. 2: How is the score normalized? Why are only four of the seven budgets shown?
  • Terminology: Please define "hazard", "deployment-time updating", and E/M/H at first use, and unify the naming of the hidden rule and the code-side predicate.

Suggestions (future work; these did not affect my score):

  • Phase boundaries are public. Removing them, which requires change-point detection, would make the benchmark more realistic.
  • Use richer environments or longer horizons with more changing phases (how does the performance change along the phases scale).
8 comments
AuthorsOpenReview ↗

Response to Reviewer fMqu (Part 1/6)

We thank the reviewer for the detailed assessment. Below, we specify the presentation revisions and report supplemental experiments on evaluation subsets examining inference delay, update representations, and safeguards. We retain the original W/Q order and label the minor questions MQ1-MQ8 in their original order.


W1: The writing and organization make the paper hard to understand, especially for readers outside this sub-area.

A1: Thank you for identifying the presentation gaps. We will make the task, information available to each agent, and execution protocol self-contained in the main text. The revisions address each of your points:

  • "Key concepts are undefined or overloaded." We will distinguish visible obstacles, rule-induced dangerous landings, executable safety predicates, and the separate HAZARD benchmark. Deployment-time updating means within-episode controller revision without weight updates; E/M/H means Easy/Medium/Hard. MQ8 gives the terminology we will use consistently.
  • "The task and the baselines are under-specified." We will state the fixed-horizon objectives in the main text and Fig. 1: crossings in Freeway and net food reward with collision penalties in Snake. World state persists across public phase boundaries while the active rule remains hidden. The installed controller acts, the Critic interprets transitions, and the coding agent proposes revisions. Q2-Q3 give the rule semantics and baseline behavior.
  • "Information the reader needs is scattered." We will put observations, catalog access, defaults, output formats, token release, continuation limits, and installation timing in one protocol description. Public boundaries and candidate catalogs are prior information; active mechanisms and parameters are hidden. Q1 specifies the additional token-accounted protocol.
  • "The presentation is not well organized or polished." We will simplify the one-use notation in Sections 3.3-3.4, reduce Fig. 4's duplication of Tables 6-7, and show raw scores for all seven budgets in Fig. 2. We will also fix Table 2 wrapping, Fig. 3 decision symbols, citation punctuation, and spacing, and replace unexplained jargon with operational definitions.
  • "Taken together, the paper reads as written for readers who already know the codebase, more like a tech note." We will organize the account as task and information, action selection, evidence-based revision, validation, then evaluation. This makes the method assessable without prior knowledge of its implementation.
AuthorsOpenReview ↗

Response to Reviewer fMqu (Part 2/6)

W2: The evaluation, as designed, does not support the "real-time" and "learning" claims.

A2: We have added evidence on charged inference delay and alternatives to source updates, and will align the claims with what each comparison establishes.

"The real-time constraint is not applied to RTHL's adaptation." The additional experiment charges generated tokens to logical actions while the installed controller continues acting. RTHL's mean scores exceed Reactive by 4.67 points on Snake and 18.33 on Freeway (Q1). This evaluates adaptation under logical inference delay; it does not retrospectively change the submitted protocol or establish uninterrupted wall-clock operation.

The "learning" is hard to separate from the engineering. The matched Full-Fixed comparison supports the contribution of online revision over the same initial policy: +10.250 points on Snake and +5.000 on Freeway (Table 3; Appendix C.4). The within-phase analysis also reports +0.197 reward per action over Fixed in the closing-minus-opening reward shift (last five versus first five phase actions; Appendix C.5, Table 18). This locates improvement within an episode.

The hand-written search, hooks, validators, and catalog remain substantial priors. To examine the update mechanism, we added a representation diagnostic: source/parameter/Fixed scores are 16/28/7 on Snake and 50/50/20 on Freeway. Parameter inference receives family schemas, ranges, periodic structure, boundary semantics, and an interpreter implementing the behaviors; coding must infer and implement compatible functions from broader descriptions. Both lack active labels and actual parameter values. The comparison shows a competitive parameter alternative, with lower implementation burden and stronger structural priors, rather than an equal-prior test of representation alone.

The non-LM comparator is also competitive (MQ4). Each diagnostic uses separately frozen controllers and prompts, with matched starting search and guards within that diagnostic; these results should not be combined with the optimized timing scores in Q1. We will frame the supported learning claim as revision of reusable behavior within this framework, without claiming source generation is uniquely necessary or superior.

"The evidence base is narrow." The transfer diagnostic yields a positive Snake example but no novel-phase gain in Freeway. Independent-environment and additional-backbone evaluations remain open. We will retain these limits and report the Freeway mean-score advantage descriptively. MQ6 provides direct evidence of rejected predictive regressions, while a separate mean-return benefit for each safeguard remains unestablished.

AuthorsOpenReview ↗

Response to Reviewer fMqu (Part 3/6)

Q1: (evaluation protocol and time accounting). The main text does not explain how RTHL's LM calls relate to the real-time clock; this has to be pieced together from App. A. Please state explicitly:

  • how many environment steps elapse while a Critic call plus a coding call is generated;
  • whether this time is charged to RTHL. Could you also report results under a protocol in which each call's generated tokens consume logical steps at the same B used for baselines, with the old controller acting in the meantime?

A1: The additional protocol charges each model call to environment actions while the installed controller continues acting. A response with L output tokens is released only after at least max(1, ceil(L/B)) intervening actions. For a Critic-plus-coding cycle, this charge applies separately to both calls and any continuations. The two roles cannot generate in the same turn, and an installed revision affects a subsequent action. Expired-phase work remains charged but cannot install in another phase.

At B=1024, RTHL/Reactive mean scores across the evaluated Easy, Medium, and Hard settings are 21.17/16.50 on Snake and 33.33/15.00 on Freeway: gains of 4.67 and 18.33 points. Matched Fixed scores are 7.00 and 21.67, respectively. All 2,280 RTHL/Reactive actions in the selected complete episodes were valid, without fallback. Two Freeway API-timeout attempts were retained before technical retries; we use the first complete attempt rather than selecting by score.

In Snake, both RTHL and Fixed include food-expiry prioritization, brief avoidance of repeated failed commands, and local movement estimation from public observations. Fixed retains these components and disables LM source revision, isolating the added revision loop over that base policy.

We will separate this additional token-accounted evaluation from the submitted blocking-call description in Appendix A. It measures logical inference delay, not uninterrupted wall-clock operation. These development subsets are not an independent test or a synchronous-versus-logical timing ablation.


Q2: (exact rule semantics). Please add a per-family table to the paper giving exact definitions and parameters. For the Snake column hazard in particular:

  • Is the whole column of each positive food lethal?
  • Does the danger follow the food as it moves, expires, or is eaten?
  • Are several columns lethal at once?
  • Is the food cell itself lethal? For Freeway, please give the lane-stay threshold, the collision-width increase, and the reversal and two-pass periods. Without these, a reader cannot judge what the agent has to learn.

A2: The food cell is safe from the extra column hazard when traversable; several other cells in positive-food columns can be lethal simultaneously. We will add the event order and per-family parameters. The settings used in the additional experiments are:

Snake: Physical collisions are checked first. On a traversable landing, food pickup occurs before the extra danger check, and any food pickup exempts that landing. Otherwise, a landing is dangerous if its x-coordinate matches any current positive food. Multiple columns can therefore be dangerous; their positions follow food movement and consumption. Food expiry occurs after this danger check, so a positive food expiring that turn still contributes to the check. Two-cell movement checks only the final landing.

Freeway: Lane entry counts zero; two subsequent consecutive stays relocate that lane's vehicles to x=4. Leaving, respawn/crossing, and phase changes reset the counter. Collision expansion applies to all eight lanes: vehicle-body overlap with [3,5] is dangerous, extending the ordinary x=4 check by one cell on each side. All-lane reversal and double traffic updates occur at positive phase-local multiples of four, with collision checked after each traffic pass. The hidden-phase local clock is zero-based.

Public phases are T10-39, T40-69, and T70-99 in Freeway, and T1-30, T31-60, and T61-90 in Snake. Exact evaluator parameters belong in the specification; they are not supplied in agent prompts.

Reviewer_uxYXOpenReview ↗
Rating
2
Confidence
3
Informativeness
3(+1)
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies adaptation to repeated hidden rule changes within continuing episodes. It introduces Evolving Freeway and Evolving Snake, where agents observe phase boundaries but must infer the active dynamics from transitions. The proposed Real-Time Heuristic Learning (RTHL) framework combines a deterministic search controller with an LM-based Critic and coding agent. The system diagnoses unexpected transitions, revises controller functions, and validates candidate changes through program checks and historical replay. Phase scoping and rollback govern subsequent reuse. Across 576 primary evaluation episodes, RTHL achieves the highest pooled mean scores at all six environment–budget settings. Matched ablations demonstrate improvements over the same initial controller without online updates. Additional experiments examine wider budgets, token usage, and within phase adaptation.

Critical strengths
  1. The investigated problem that studies adaptation to repeated hidden rule changes within continuing episodes is very interesting, and the writing is easy to read and understand.

  2. The benchmark tests adaptation to hidden rule changes while preserving world state across phases.

  3. The matched ablation shows that online updates improve performance beyond the same initial controller.

Critical weaknesses
  1. The evaluation does not include real time adaptation. Critic and coding calls block environment advancement, so the experiments do not test whether updates arrive quickly enough while the world continues evolving in Section 2.2.

  2. The comparisons do not isolate the benefit of code revision. From Table 3 and Section 4.2, the Fixed baseline disables all online updates, and the paper lacks a comparison with the same controller receiving simpler rule or parameter updates.

  3. The ablations provide limited evidence for the safeguards’ benefits. All reported paired confidence intervals for removing replay/rollback or phase scoping include zero, and removing replay/rollback slightly improves Snake’s mean score.

Questions for authors
  1. How does RTHL perform when the environment advances during Critic and coding calls, while the installed controller continues acting until a revision finishes?

  2. How does RTHL compare with the same search controller receiving structured rule or parameter updates from an inference module, without generating new source code?

  3. What measurable reliability benefit do replay/rollback and phase scoping provide?

4 comments
AuthorsOpenReview ↗

Response to Reviewer uxYX (Part 1/3)

We thank the reviewer for the concrete suggestions. We added a token-accounted evaluation and a parameter-update comparison on evaluation subsets, alongside an audit of actual model proposals. Below, we address each weakness and question with the corresponding evidence.


W1: The evaluation does not include real time adaptation. Critic and coding calls block environment advancement, so the experiments do not test whether updates arrive quickly enough while the world continues evolving in Section 2.2.

A1: We added the requested evaluation with inference tokens charged to environment actions. The installed controller continues acting until a response is released; a revision affects only subsequent actions. At B=1024, RTHL's mean score exceeds Reactive by 4.67 points on Snake and 18.33 on Freeway in the additional development evaluation. Q1 gives the accounting rule and results. We will distinguish this logical-time protocol from the submitted blocking-call description and from uninterrupted wall-clock operation.


W2: The comparisons do not isolate the benefit of code revision. From Table 3 and Section 4.2, the Fixed baseline disables all online updates, and the paper lacks a comparison with the same controller receiving simpler rule or parameter updates.

A2: We added a same-search parameter-update comparator. It scores 28 versus 16 for source updates on Snake and ties at 50 on Freeway in the representation diagnostic (Q2). These results directly address the missing comparator. We agree that Full versus Fixed isolates the whole revision loop; it cannot establish an advantage of source generation over parameter updates. We will report the competitive parameter results together with their stronger structural priors and preimplemented interpreter.


W3: The ablations provide limited evidence for the safeguards’ benefits. All reported paired confidence intervals for removing replay/rollback or phase scoping include zero, and removing replay/rollback slightly improves Snake’s mean score.

A3: The additional audit shows that replay can prevent observed prediction regressions from being installed. Five rejected proposals would have replaced a correct prediction with an incorrect one on recorded transitions; Q3 gives the audit and controlled fault checks. This is a concrete reliability benefit. The return ablations remain inconclusive about a separate mean-return benefit for each safeguard: Full minus No Replay/Rollback is -0.875 points on Snake (Appendix C.4). We will distinguish these two findings.

AuthorsOpenReview ↗

Response to Reviewer uxYX (Part 2/3)

Q1: How does RTHL perform when the environment advances during Critic and coding calls, while the installed controller continues acting until a revision finishes?

A1: RTHL achieves higher mean scores than Reactive and matched Fixed under the additional token-accounted protocol. At B=1024, the means across the evaluated Easy, Medium, and Hard settings are:

\hline \text{Game} & \text{RTHL} & \text{Reactive} & \text{Matched Fixed} & \text{RTHL - Reactive} \\\\ \hline \text{Snake} & 21.17 & 16.50 & 7.00 & +4.67 \\\\ \text{Freeway} & 33.33 & 15.00 & 21.67 & +18.33 \\\\ \hline \end{array}$$ **How inference consumes time:** A response containing L output tokens is released only after at least max(1, ceil(L/B)) intervening actions by the installed controller. Continuations are charged separately, Critic and coding cannot generate in the same turn, and installation affects a subsequent action. Work completing after its phase is still charged but cannot install in another phase. Thus inference can consume the time available for useful adaptation while the existing policy continues acting. All **2,280** RTHL/Reactive actions in the selected complete episodes were valid, without fallback. Two Freeway API-timeout attempts were retained before technical retries; we use the first complete attempt rather than selecting by score. The matched Snake Fixed controller includes the same food-expiry prioritization, brief avoidance of repeated failed commands, and movement estimation from public observations as RTHL. Fixed retains this empirical adaptation and disables LM source revision, so the comparison measures the added revision loop over the same base policy. We will report this as an additional logical-time evaluation, separate from the blocking-call description in Appendix A. These cases informed development; the results are descriptive rather than an independent test, an uninterrupted wall-clock evaluation, or a synchronous-versus-logical timing ablation.
AuthorsOpenReview ↗

Response to Reviewer uxYX (Part 3/3)

Q2: How does RTHL compare with the same search controller receiving structured rule or parameter updates from an inference module, without generating new source code?

A2: Parameter updates score higher on Snake and tie source updates on Freeway in this diagnostic. The methods share starting search, model settings, public transitions, scheduling, and validation. Controller and role-specific prompt configurations were frozen separately from the main timing experiment, so the following scores should be compared within this diagnostic:

\hline \text{Game} & \text{Source updates} & \text{Parameter updates} & \text{Fixed} \\\\ \hline \text{Snake} & 16 & 28 & 7 \\\\ \text{Freeway} & 50 & 50 & 20 \\\\ \hline \end{array}$$ **The information and implementation requirements differ.** Parameter inference receives family names, parameter fields and ranges, a periodic functional form, and some boundary semantics. Its fixed interpreter already implements the candidate behaviors. The coding agent receives broad descriptions and must infer the behavior and implement functions satisfying interface, runtime, and replay requirements. Neither method receives the active family or its actual parameter values. The parameter approach therefore has more explicit structural information and less implementation work. We will state this difference alongside the competitive results: this is not an equal-prior, representation-only ablation. Source code offers an extensible executable representation, but these scores establish neither its necessity nor its superiority. Equal per-turn B also does not imply equal total tokens or input information. --- **Q3:** What measurable reliability benefit do replay/rollback and phase scoping provide? **A3:** **Historical replay rejected five proposals with observed predictive regressions.** Among 14 actual model proposals reaching replay in two development episodes, eight passed and six were rejected. Five rejected candidates had a recorded transition on which the installed predictor was correct and the candidate was wrong. All six had passed sandbox contracts, so removing only replay would admit those same candidates at the saved states. For example, a Snake candidate inverted food values on a transition whose layout and values the installed predictor already matched. Replay rejected this behavioral error even though the candidate passed the program contracts. Separately, **36 deterministic fault probes** (two games, six faults, three guard settings) verify rejection of historical contradictions, recovery after two supported contradictory observations, and deactivation at phase boundaries. Removing the corresponding safeguard exposes the specified fault. These checks distinguish replay before installation, rollback after supported contradictions, and phase scoping when a regime ends. We will report these reliability findings separately from return effects. The submitted Snake Full-minus-No-Replay/Rollback difference is -0.875 points (Appendix C.4), so a separate mean-return gain is not established. The saved-proposal audit and controlled faults also do not estimate a population-level harmful-installation rate or replace a closed-loop return ablation.
Reviewer_uxYXOpenReview ↗

Response to Rebuttal

Thanks for the detailed rebuttal. However, considering the remaining questions about whether revisions can be installed in time to help under actual inference latency, whether source generation adds value over structured parameter updates, as well as the remained concerns by reviewer fMqu, I will keep my score at 2. I will raise the informative score to 3 to reflect the effort of the rebuttal.

Reviewer_GYtPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces RTHL, an evidence-to-code framework compiling transition evidence into reusable executable rules, which perform good on extended Real-Time Reasoning Gym without LLM reasoning.

Critical strengths
  1. The paper identifies a useful distinction between adaptation to changing state and adaptation to changing transition or reward rules. Public boundaries, hidden mechanisms, and persistent state make the resulting evaluation more meaningful than simply restarting an episode after each change, and the logged execution cases make the intended evidence-to-policy loop concrete.

  2. The comparison to the identical fixed initial controller is a strong piece of evidence for the core claim that online revision adds value: the reported paired gains are substantial and their confidence intervals exclude zero in both games. The paper is also unusually explicit about seeds, variance, installation timing, and resource-accounting limitations, which makes the empirical claims easier to audit.

Critical weaknesses

External validity and attribution are limited. Both environments and the method are developed together, and RTHL begins from a hand-written, game-specific search controller with editable prediction and safety hooks closely matched to the benchmark's rule changes; two small grid games therefore do not establish that the approach generalizes to less structured or independently designed settings.

Questions for authors

[Suggestion] Can the authors evaluate transfer to an independently designed environment or to held-out rule families that were not anticipated by the controller hooks and validation logic? A positive result would substantially strengthen the generality claim.

1 comment
AuthorsOpenReview ↗

Response to Reviewer GYtP

We thank the reviewer for the constructive feedback and for recognizing the value of the matched-controller comparison. We clarify that evidence and report the requested additional transfer diagnostic on an exploratory evaluation subset.


W1: External validity and attribution are limited. Both environments and the method are developed together, and RTHL begins from a hand-written, game-specific search controller with editable prediction and safety hooks closely matched to the benchmark's rule changes; two small grid games therefore do not establish that the approach generalizes to less structured or independently designed settings.

A1: Thank you for highlighting the distinction between attribution and generalization. The matched-controller experiment supports a contribution from online revision beyond the starting controller. Full and Fixed share the same initial controller and memory; enabling the revision loop yields gains of 10.250 points on Snake and 5.000 on Freeway (Table 3; Appendix C.4), as you also recognize in the strengths.

The hand-written search and editable prediction/safety interfaces remain substantial design priors. Accordingly, we will state the contribution as evidence-driven revision of reusable behavior within this framework. The matched comparison attributes gains to the revision loop as a whole; it does not isolate source generation or establish generalization to independently designed environments. Our answer to Q1 tests transfer beyond the supplied rule catalog.


Q1: [Suggestion] Can the authors evaluate transfer to an independently designed environment or to held-out rule families that were not anticipated by the controller hooks and validation logic? A positive result would substantially strengthen the generality claim.

A1: We added a transfer test with mechanisms absent from the supplied catalog, retaining the existing hooks and validators. Snake uses quarter-turn command remapping and toroidal food translation; Freeway uses alternating lane updates and positional mirroring without direction reversal. These change the transition mechanism itself. Each episode contains two novel phases and one familiar calibration phase, and the agent receives no implementation of the new mechanisms.

Controller, prompt, and validation settings were frozen before these model runs. RTHL and Fixed share the initial search. This diagnostic uses a separate controller/prompt configuration from the main timing comparison. At B=1024 under token accounting, the scores are as follows; each triple is ordered (RTHL, Reactive, Fixed).

\hline \text{Game} & \text{Whole-episode score} & \text{Reward in two novel phases} \\\\ \hline \text{Snake} & (26, 26, 5) & (20, 18, -2) \\\\ \text{Freeway} & (50, 40, 60) & (-30, -20, -20) \\\\ \hline \end{array}$$ **Definition of phase reward:** The last column sums reward increments over the 60 actions in the two novel phases. In Freeway, each action costs one point and each successful crossing earns ten points, so $R_{novel} = 10 C_{novel} - 60$, where $C_{novel}$ denotes the number of successful crossings within those phases. Thus three crossings give $30 - 60 = -30$. This subtotal excludes the initial 100 points and all other phases; the whole-episode score includes that initial balance and reward increments from all 100 actions. **Snake shows adaptation beyond the supplied catalog.** Movement and food revisions became effective at actions 8 and 74 in the corresponding novel phases. RTHL tied Reactive overall and improved over Fixed; its novel-phase reward was 20, compared with 18 and -2. **Freeway did not show successful novel-rule adaptation.** It installed a revision only in the familiar calibration phase and underperformed both comparators during the novel phases. Its higher whole-episode score than Reactive therefore cannot be attributed to adapting to the new mechanisms. We will report both outcomes as exploratory transfer evidence. The mechanisms remain developer-designed within the same games, so this test does not substitute for an independently designed environment or prove that the interface designers had never anticipated them. We will also distinguish executing a revision successfully from identifying the complete hidden rule.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.