- Rating
- 1
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This work focuses on the interesting topic of Heuristic Learning, proposing one benchmark and one scaffold. The benchmark extends Real-Time Reasoning Gym from changing states to changing rules, with two environments: Evolving Freeway and Evolving Snake. Each episode has three 30-action phases. The phase boundaries are public, the active mechanism is hidden, and world state persists across boundaries.
The proposed method, RTHL, uses code as an evolving harness:
- A deterministic controller selects every action.
- An LM Critic diagnoses evidence after each transition.
- A coding agent then installs validated, phase-scoped revisions to that controller.
The experiments cover:
- a main comparison across both games, three difficulty levels, and several token budgets;
- a matched ablation against the same controller with online updates disabled;
- component ablations of replay/rollback and phase scoping;
- an extended budget sweep on the Hard setting;
- token-usage accounting.
RTHL outperforms three reasoning baselines (Reactive, Planning, and AgileThinker), with the clearest advantage on Snake. It also improves over its own fixed starting controller, and it generates less LM output than the baselines. A within-phase reward-shift analysis and installation-timing traces show adaptation happening within a regime.
Critical strengths
- A well-motivated problem and a natural design answer to it.
- The paper identifies a real gap in existing real-time agent benchmarks. They test whether an agent can react to changing states, but not whether it can discover and reuse changing rules within one continuing episode (§1; App. D, Table 20).
- The two Evolving environments target exactly this gap. Rules switch while the world state persists, and the agent must infer the switch from outcomes alone. This is a useful extension of Real-Time Reasoning Gym (§2.1, Fig. 1, Table 1; App. A).
- The scaffold separates acting (an installed deterministic controller) from revising the policy (Critic diagnosis followed by validated code edits; §3, Fig. 3). Existing approaches (Reactive, Planning, AgileThinker) carry what they discover forward only as prompt text, queued actions, or textual guidance (§3.1, Table 2). RTHL instead turns a discovery into code that is re-applied to every new state and switched off when its phase ends (§3.2–3.4; Eqs. 1–5).
- This removes the LM from the per-step action path. That is a sensible response to the latency-versus-quality tension that motivates real-time reasoning benchmarks, and it also reduces how much LM output is generated per episode compared with the baselines (§4.3, Table 4; App. C.3).
- Controlled experiments that attribute the gain to online revision and locate it in time.
- A matched ablation keeps exactly the same starting controller and memory and only disables the LM revision loop. Full RTHL beats this fixed version in both games and in every difficulty setting (§4.2, Table 3; App. C.4, Table 17).
- The fixed controller on its own does not beat the strongest baseline. So the advantage over existing methods comes from the online updates, not simply from the hand-written controller (App. C.4, Table 17 vs. App. C.1, Tables 6–7).
- The advantage over the three baselines is consistent on Snake across difficulty levels and token budgets. It holds even at large budgets where the Planning baseline finally produces usable plans, and it grows as the navigation constraints get harder (§4.1, Fig. 4; App. C.1, Table 7; §4.3, Fig. 5; App. C.2, Tables 8 and 10).
- The paper goes beyond final scores:
- A within-phase analysis compares reward at the start and end of each rule phase against the fixed controller. It shows that performance improves within a regime, not merely across episodes (§4.2; App. C.5, Eq. 6, Table 18).
- Installation-timing traces show that most phases receive a usable code revision early enough to be reused before the rules change again (App. C.5).
- Logged execution cases trace a single diagnosis through validation, installation, reuse on later states, and deactivation at the phase boundary. This makes the mechanism concrete and auditable (App. E, Tables 21–22).
Critical weaknesses
- The writing and organization make the paper hard to understand, especially for readers outside this sub-area. This is my main concern, and it affects how far the results can be checked at all. (The full itemized list is in the Presentation field.)
- Key concepts are undefined or overloaded.
- "Hazard" is never defined, yet it is used for four different things: the ordinary word, the HAZARD benchmark, one of the paper's hidden rules, and a code hook. The paper also lists obstacles and hazards side by side in §2.1 without explaining the difference. A reader only learns that hazards are hidden lethal cells from an execution case on p.24.
- "Deployment-time updating" (a stated contribution, l.121) is never defined.
- The abbreviation E/M/H is used without being introduced.
- The task and the baselines are under-specified.
- The hidden rules are described in a few words each, for example "make positive-food columns dangerous". From this, the reader cannot tell what an agent must actually learn, or how hard each rule is.
- Reactive, Planning, and AgileThinker each get one or two compressed sentences (§3.1, Table 2). As a result, it is hard to interpret their results, for example why Planning scores strongly negative on Snake.
- Fig. 1 does not even state the objective, which is to maximize crossings or net food reward within a fixed horizon.
- Information the reader needs is scattered. Facts required to read the headline results sit in appendix paragraphs or are missing entirely:
- that RTHL's LM calls are not charged to the real-time clock (App. A, "Temporal Execution");
- that all agents are given a catalog of rule families (App. A);
- how Fig. 2 is normalized (not stated). Meanwhile, §3.3–3.4 introduce a dense set of symbols that are mostly used once, and Fig. 4 spends most of a page re-plotting Tables 6–7.
- The presentation is not well organized or polished.
- Table 2 entries break mid-word.
- Fig. 3 mixes text labels and decision diamonds for its branches.
- Citations are not parenthesized, and there are many spacing errors.
- The prose relies heavily on unexplained jargon ("architecture-native", "logical release", "action authority").
- Taken together, the paper reads as written for readers who already know the codebase, more likely a tech note. The idea itself is not complicated, but a reader unfamiliar with Real-Time Reasoning Gym or this domain cannot reconstruct the task, the baselines, or the evaluation protocol from the main text.
- The evaluation, as designed, does not support the "real-time" and "learning" claims.
- The real-time constraint is not applied to RTHL's adaptation.
- In the baselines, the token budget B determines which reasoning is available at each decision point.
- For RTHL, B only caps each Critic or coding call, and continuation is allowed. Its adaptation "completes before the next environment step" (l.151–152), and its LM calls "block advancement of the Python loop" (App. A).
- So RTHL diagnoses and rewrites code while the world is effectively paused. This is exactly the pause-during-deliberation setting the paper criticizes in WebArena/KORGym (l.454).
- A symptom: Snake-H RTHL scores are identical at six of seven budgets (Table 8).
- A fair protocol would charge each call's generated tokens to logical time, with the old controller acting in the meantime.
- The "learning" is hard to separate from the engineering.
- RTHL starts from a hand-written, environment-specific search controller (App. B).
- Edits are restricted to named hooks that map almost one-to-one onto the implemented rule families.
- Each hook has hand-tuned validators.
- The catalog of rule families is supplied to the agent (App. A).
- The task is therefore closer to identifying which known mechanism is active and filling a pre-built slot than to compiling open-ended evidence into code.
- The most informative baseline is missing: the same controller plus a simple non-LM consistency test over the catalog.
- The evidence base is narrow.
- Every LM role uses a single model, DeepSeek-V4-Flash.
- The Freeway advantage over Reactive is within noise.
- The replay/rollback and phase-scoping components show no measurable effect (Table 17).
Questions for authors
Q1 (evaluation protocol and time accounting). The main text does not explain how RTHL's LM calls relate to the real-time clock; this has to be pieced together from App. A. Please state explicitly:
- how many environment steps elapse while a Critic call plus a coding call is generated;
- whether this time is ever charged to RTHL. Could you also report results under a protocol in which each call's generated tokens consume logical steps at the same B used for the baselines, with the old controller acting in the meantime?
Q2 (exact rule semantics). Please add a per-family table to the paper giving exact definitions and parameters. For the Snake column hazard in particular:
- Is the whole column of each positive food lethal?
- Does the danger follow the food as it moves, expires, or is eaten?
- Are several columns lethal at once?
- Is the food cell itself lethal? For Freeway, please give the lane-stay threshold, the collision-width increase, and the reversal and two-pass periods. Without these, a reader cannot judge what the agent has to learn.
Q3 (baselines). Please describe concretely how Reactive, Planning, and AgileThinker work in your setting:
- the prompt and output format; the release mechanism under B; the default action in each game; how AgileThinker's fast branch uses the slow trace; how the three differ from each other.
- Relatedly, is the default action itself lethal in Snake? This would explain Planning's strongly negative scores (Tables 7, 19).
- Evaluate with additional backbone LMs. A cheap but strong model (e.g., GPT-6-Luna) and a small open model (e.g., Qwen3.8-27B) would test whether RTHL degrades more gracefully than Reactive as the LM becomes weaker.
Minor questions:
- Budget invariance: Why are the Snake-H RTHL scores identical at six of seven budgets (Table 8), while Freeway-H varies non-monotonically (Table 9)? What sampling temperature does each role use?
- Freeway significance: Please report paired CIs for RTHL vs Reactive in each cell (Table 6), as you do for Full vs Fixed.
- Identification accuracy: From your logs, can you report how often the installed hook matches the true active rule, the false-install rate, and a per-family breakdown?
- Non-LM baseline: How would the same controller with a hand-coded consistency test over the known rule families perform?
- Seed selection: How were the non-contiguous Freeway seeds chosen? Were the controllers, hooks, and validators developed on these same seeds?
- Guarded consolidation: Replay/rollback and phase scoping show no score effect (Table 17). Do they reduce harmful installs or variance?
- Fig. 2: How is the score normalized? Why are only four of the seven budgets shown?
- Terminology: Please define "hazard", "deployment-time updating", and E/M/H at first use, and unify the naming of the hidden rule and the code-side predicate.
Suggestions (future work; these did not affect my score):
- Phase boundaries are public. Removing them, which requires change-point detection, would make the benchmark more realistic.
- Use richer environments or longer horizons with more changing phases (how does the performance change along the phases scale).
8 comments
Response to Reviewer fMqu (Part 2/6)
W2: The evaluation, as designed, does not support the "real-time" and "learning" claims.
A2: We have added evidence on charged inference delay and alternatives to source updates, and will align the claims with what each comparison establishes.
"The real-time constraint is not applied to RTHL's adaptation." The additional experiment charges generated tokens to logical actions while the installed controller continues acting. RTHL's mean scores exceed Reactive by 4.67 points on Snake and 18.33 on Freeway (Q1). This evaluates adaptation under logical inference delay; it does not retrospectively change the submitted protocol or establish uninterrupted wall-clock operation.
The "learning" is hard to separate from the engineering. The matched Full-Fixed comparison supports the contribution of online revision over the same initial policy: +10.250 points on Snake and +5.000 on Freeway (Table 3; Appendix C.4). The within-phase analysis also reports +0.197 reward per action over Fixed in the closing-minus-opening reward shift (last five versus first five phase actions; Appendix C.5, Table 18). This locates improvement within an episode.
The hand-written search, hooks, validators, and catalog remain substantial priors. To examine the update mechanism, we added a representation diagnostic: source/parameter/Fixed scores are 16/28/7 on Snake and 50/50/20 on Freeway. Parameter inference receives family schemas, ranges, periodic structure, boundary semantics, and an interpreter implementing the behaviors; coding must infer and implement compatible functions from broader descriptions. Both lack active labels and actual parameter values. The comparison shows a competitive parameter alternative, with lower implementation burden and stronger structural priors, rather than an equal-prior test of representation alone.
The non-LM comparator is also competitive (MQ4). Each diagnostic uses separately frozen controllers and prompts, with matched starting search and guards within that diagnostic; these results should not be combined with the optimized timing scores in Q1. We will frame the supported learning claim as revision of reusable behavior within this framework, without claiming source generation is uniquely necessary or superior.
"The evidence base is narrow." The transfer diagnostic yields a positive Snake example but no novel-phase gain in Freeway. Independent-environment and additional-backbone evaluations remain open. We will retain these limits and report the Freeway mean-score advantage descriptively. MQ6 provides direct evidence of rejected predictive regressions, while a separate mean-return benefit for each safeguard remains unestablished.
Response to Reviewer fMqu (Part 3/6)
Q1: (evaluation protocol and time accounting). The main text does not explain how RTHL's LM calls relate to the real-time clock; this has to be pieced together from App. A. Please state explicitly:
- how many environment steps elapse while a Critic call plus a coding call is generated;
- whether this time is charged to RTHL. Could you also report results under a protocol in which each call's generated tokens consume logical steps at the same B used for baselines, with the old controller acting in the meantime?
A1: The additional protocol charges each model call to environment actions while the installed controller continues acting. A response with L output tokens is released only after at least max(1, ceil(L/B)) intervening actions. For a Critic-plus-coding cycle, this charge applies separately to both calls and any continuations. The two roles cannot generate in the same turn, and an installed revision affects a subsequent action. Expired-phase work remains charged but cannot install in another phase.
At B=1024, RTHL/Reactive mean scores across the evaluated Easy, Medium, and Hard settings are 21.17/16.50 on Snake and 33.33/15.00 on Freeway: gains of 4.67 and 18.33 points. Matched Fixed scores are 7.00 and 21.67, respectively. All 2,280 RTHL/Reactive actions in the selected complete episodes were valid, without fallback. Two Freeway API-timeout attempts were retained before technical retries; we use the first complete attempt rather than selecting by score.
In Snake, both RTHL and Fixed include food-expiry prioritization, brief avoidance of repeated failed commands, and local movement estimation from public observations. Fixed retains these components and disables LM source revision, isolating the added revision loop over that base policy.
We will separate this additional token-accounted evaluation from the submitted blocking-call description in Appendix A. It measures logical inference delay, not uninterrupted wall-clock operation. These development subsets are not an independent test or a synchronous-versus-logical timing ablation.
Q2: (exact rule semantics). Please add a per-family table to the paper giving exact definitions and parameters. For the Snake column hazard in particular:
- Is the whole column of each positive food lethal?
- Does the danger follow the food as it moves, expires, or is eaten?
- Are several columns lethal at once?
- Is the food cell itself lethal? For Freeway, please give the lane-stay threshold, the collision-width increase, and the reversal and two-pass periods. Without these, a reader cannot judge what the agent has to learn.
A2: The food cell is safe from the extra column hazard when traversable; several other cells in positive-food columns can be lethal simultaneously. We will add the event order and per-family parameters. The settings used in the additional experiments are:
Snake: Physical collisions are checked first. On a traversable landing, food pickup occurs before the extra danger check, and any food pickup exempts that landing. Otherwise, a landing is dangerous if its x-coordinate matches any current positive food. Multiple columns can therefore be dangerous; their positions follow food movement and consumption. Food expiry occurs after this danger check, so a positive food expiring that turn still contributes to the check. Two-cell movement checks only the final landing.
Freeway: Lane entry counts zero; two subsequent consecutive stays relocate that lane's vehicles to x=4. Leaving, respawn/crossing, and phase changes reset the counter. Collision expansion applies to all eight lanes: vehicle-body overlap with [3,5] is dangerous, extending the ordinary x=4 check by one cell on each side. All-lane reversal and double traffic updates occur at positive phase-local multiples of four, with collision checked after each traffic pass. The hidden-phase local clock is zero-based.
Public phases are T10-39, T40-69, and T70-99 in Freeway, and T1-30, T31-60, and T61-90 in Snake. Exact evaluator parameters belong in the specification; they are not supplied in agent prompts.
Response to Reviewer fMqu (Part 1/6)
We thank the reviewer for the detailed assessment. Below, we specify the presentation revisions and report supplemental experiments on evaluation subsets examining inference delay, update representations, and safeguards. We retain the original W/Q order and label the minor questions MQ1-MQ8 in their original order.
W1: The writing and organization make the paper hard to understand, especially for readers outside this sub-area.
A1: Thank you for identifying the presentation gaps. We will make the task, information available to each agent, and execution protocol self-contained in the main text. The revisions address each of your points: