acceptodds
Under review as a conference paper at ICLR 2027

Learning Reachability Rewards for Temporal-Logic Reinforcement Learning

Abstract

Reinforcement learning with Linear Temporal Logic (LTL) specifications faces severe credit-assignment challenges when learning infinite-horizon Büchi objectives. We propose RACE (Reachability-Aware Credit Estimation), a framework that uses accepting-set reachability as a surrogate for learning recurrent acceptance. RACE trains a discriminator on trajectories collected by the agent to predict whether the rollout continuation from a sampled state reaches acceptance. Its predictions provide dense, state-dependent rewards for policy learning. We extend this construction to intermediate stages of the limit-deterministic Büchi automaton (LDBA), allowing partially successful trajectories to provide useful feedback even when they do not reach acceptance. Automaton-induced potentials complement these learned rewards by guiding exploration. Our analysis establishes a scaled lower bound on an infinite-time reachability objective and provides conditional connections to Büchi satisfaction. Empirically, RACE improves exploration efficiency and task performance across continuous-control domains with complex LTL objectives.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Temporal Logic

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mtfmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RACE for reinforcement learning under infinite-horizon LTL specifications, using a product of the environment and a limit-deterministic Buchi automaton. It supports continuous environment states and actions, with discrete automaton choices where enabled. Component-wise classifiers turn acceptance and partial progress in agent-generated trajectories into dense rewards, combined with automaton-based exploration potentials and SAC policy updates. The theory establishes a finite-rollout reachability lower bound and conditional links between an idealized objective and recurrent acceptance, rather than optimality of the learned SAC policy. Experiments compare RACE with LTL baselines across navigation, manipulation, and locomotion, including ablations and extended-horizon execution tests.

Critical strengths

The method extracts learning signals from partially successful trajectories without demonstrations. The experiments cover several domains, include component ablations and extended-horizon accepting-round counts, and report gains over relevant LTL baselines. The analysis explicitly distinguishes its conditional surrogate results from guarantees for the implemented learner.

Critical weaknesses

The evaluation protocol needs reconciliation. Appendix H specifies 50-step Fetch episodes, while Figure 12 reports horizons of 100 for RACE and 50 for TiLoIL. The evaluation discount or scaling also needs clarification: Figure 4 contains returns above 100, whereas the unit-reward metric in Appendix P.2 would be bounded by 100 if it used the reported training discount of 0.99. These may be reporting differences, but they currently limit interpretation of the comparisons.

Equation 21 also assigns the same Q target to every action at accepting states. In the ideal critic limit, Equation 22 then supplies only an entropy incentive at those states. How safe, recurrent continuation is learned under this boundary rule remains unclear.

Questions for authors
  1. Continuation after acceptance. Equation 21 gives every action at an accepting state the same target C. In the ideal critic limit, Equation 22 therefore leaves only an entropy incentive there. How does the actor learn to choose safe actions that begin subsequent rounds? Safety-violation rates and successful continuation rates would help interpret the accepting-round counts in Appendix M.
  2. Connection between theory and implementation. The conditional optimality result concerns a global optimizer of an idealized objective, while the practical heads use balanced, pooled whole-segment labels (Appendices C.5 and G). Could a calibration check against current-policy future-progress probabilities, or a matched-suffix-label ablation, show how closely the practical rewards reflect the reachability motivation?
  3. Continuous and discrete actions. How are enabled discrete automaton epsilon-actions selected alongside continuous SAC controls? Do any evaluated tasks require these choices, and what policy architecture or selection rule handles them?
2 comments

Thank you for the questions on post-acceptance control and discriminator validation. We address each below.

Continuation after acceptance

Equation 21 gives every action at an accepting state the same target CC. In the ideal critic limit, Equation 22 therefore leaves only an entropy incentive there. How does the actor learn to choose safe actions that begin subsequent rounds? Safety-violation rates and successful continuation rates would help interpret the accepting-round counts in Appendix M.

Response. Our implementation uses the action-dependent target in Appendix D.1 and Algorithm 1. The constant CC supplies completion credit on arrival at acceptance; actions taken from accepting states are evaluated through their immediate rewards and successor states. The actor therefore receives task-dependent action values in addition to the entropy incentive. The uniform boundary Q(x,a)=CQ(x,a)=C simplifies the theoretical analysis.

With C=1/(1−γ)C=1/(1-\gamma) and structural failure set F⊥F_\bot, the implemented target is

\widehat y&=R(x,x')+\gamma\widetilde V(x'),\\ \widetilde V(z)&= \begin{cases} C, & z\in F,\\ 0, & z\in F_\bot,\\ \mathbb E_{a\sim\pi(\cdot\mid z)}[Q_{\bar\theta}(z,a)-\alpha\log\pi(a\mid z)], & \text{otherwise}. \end{cases} \end{aligned}$$ Every action, including one selected at acceptance, is evaluated through its own reward and successor state. Entering structural failure retains the immediate reward but receives no future bootstrap; continuing toward another accepting visit retains the corresponding future credit. D.2 gives the explicit task-value comparison, which applies without entropy. In D.4’s exact, horizon-consistent reachability construction, outgoing action values are proportional to the probability of another accepting visit. The critic therefore provides action-dependent feedback for the next segment. We evaluate policies using this implemented target at $H$, $5H$, and $10H$ without further training or resets at acceptance, reporting accepting visits and post-acceptance violations separately. **Accepting visits at the original and extended horizons.** Accepting-visit entries are mean ± SD. | Environment (H steps) | H | 5H | 10H | Post-acceptance violation: H / 5H / 10H | |---|---:|---:|---:|---| | Carlo (500) | 2.40 ± 1.27 | 11.83 ± 14.59 | 20.67 ± 32.06 | 25.00% / 75.00% / 75.00% | | Cheetah Flip (1,000) | 14.63 ± 9.47 | 21.34 ± 21.49 | 21.34 ± 21.49 | N/A | | FlatWorld Cycle (50) | 3.33 ± 1.49 | 5.50 ± 9.95 | 11.25 ± 20.35 | 0.00% / 0.00% / 0.00% | | SixteenRooms (1,000) | 38.67 ± 33.49 | 195.33 ± 169.16 | 391.33 ± 338.90 | 0.00% / 0.00% / 0.00% | | PointZone (1,000) | 2.20 ± 2.05 | 12.00 ± 11.02 | 24.00 ± 22.05 | 0.00% / 0.00% / 0.00% | | FetchPlaceButton (500) | 1.97 ± 2.39 | 6.98 ± 12.33 | 13.23 ± 24.82 | N/A | Seed counts are 4/3/10/3/5/4, for Carlo/ Cheetah/ FlatWorld/ SixteenRooms/ PointZone/ FetchPlaceButton, respectively. Frozen deterministic policies run without learning or resets at acceptance, with 100 episodes per seed; except PointZone, which uses one deterministic trajectory per seed. Each rollout supplies the H/5H/10H prefixes. Visits are reported as mean ± sample SD across seed means. Post-acceptance violation rates measure the fraction of accepting episodes whose first violation follows first acceptance, averaged equally across seeds. N/A indicates no safety clause. Mean accepting visits increase beyond the original horizon in all six recurrent benchmarks, with approximately proportional growth in SixteenRooms and PointZone. These tasks and FlatWorld Cycle show no observed post-acceptance violations. Cheetah Flip achieves approximately 46% more accepting visits at $5H$ under demanding six-degree-of-freedom flip dynamics. Carlo achieves 8.6 times as many accepting visits at $10H$, with further growth from $5H$ to $10H$ while the observed post-acceptance violation rate remains unchanged. These results provide direct evidence of continued execution without additional training.

Connection between theory and implementation

We directly test whether practical discriminator scores predict subsequent progress and acceptance under the current policy. At the final training checkpoint, we freeze the policy and discriminators and collect 100 fresh stochastic episodes per seed at horizon HH for each reported task, using two training seeds. Each discriminator is evaluated within its own nonaccepting component, with labels based on strictly subsequent outcomes. We assess ranking with AUROC and calibration with expected calibration error (ECE).

These diagnostics provide replicated evidence in Doggo Navigate: both component-wise discriminators give informative progress and acceptance rankings in both training seeds. Doggo Avoid and Carlo provide additional progress-ranking evidence. The table below summarizes these diagnostics.

Final-checkpoint diagnostics for tasks with defined progress AUROC. Ranges cover visited component–seed pairs; parentheses give the number with defined AUROC. “One class” denotes undefined AUROC.

EnvironmentProgress AUROC ↑Acceptance AUROC ↑Progress ECE ↓
Doggo Navigate0.734–0.986 (4)0.734–0.904 (4)0.068–0.209
Doggo Avoid0.949 (1)0.949 (1)0.293–0.397
Carlo0.910 (1)One class<0.001 to 0.363

These measurements complement Proposition 4 in Appendix C.5, which characterizes the discrepancy between the learned reward objective and current-policy progress, accounting for fitting error and distribution mismatch. Figure 7's component ablations provide complementary evidence for the contribution of discriminator feedback to learning performance.

Evaluation protocol and reported returns

Figure 4 reports raw acceptance returns: the undiscounted sum of unit acceptance rewards over an episode. The reported γ=0.99\gamma=0.99 is the SAC training discount; the plotted raw return is bounded by the episode horizon HH. This explains returns above 100 in Doggo's 500-step episodes and returns near 90 in RACE's 100-step Fetch episodes. We have corrected the Figure 4 caption, the main results description, and Appendix P.2 to state the evaluation metric explicitly.

Our new TiLoIL and TiLoIL-plus-potential Fetch runs both use H=100H=100, matching RACE, at matched online interaction budgets. This replaces the H=50H=50 TiLoIL setting in the original Figure 12 comparison. The two TiLoIL variants use the same demonstrations and tuning settings; RACE uses no demonstrations. Both TiLoIL variants use the same raw acceptance-return definition as RACE. The results are included below:

Fetch comparison at H=100H=100 and matched online budgets.

TaskOnline stepsRACE referenceTiLoILTiLoIL + potential
Fetch Avoid400K91.79 ± 1.7776.31 ± 5.4361.49 ± 18.48
Fetch Align2M47.82 ± 35.0733.40 ± 35.5136.87 ± 35.08

We have updated Appendix H and the Figure 12 description to state the horizons for the original and new comparisons explicitly.

Continuous controls and discrete automaton actions

Our implementation uses a learned hybrid policy to select between physical control and automaton epsilon transitions. A masked categorical selector chooses either the physical-action branch or an enabled epsilon edge. On the physical branch, a tanh-Gaussian policy supplies the continuous action. The selector and continuous controller are trained jointly using action values and entropy terms, with invalid epsilon choices masked. The policy therefore learns both when to commit and which enabled transition to take.

An epsilon action changes only the automaton state. It does not move the simulator, advance physical time, consume a new physical label, or receive a physical accepting-visit reward. Subsequent physical actions are evaluated under the new automaton state.

The compiled automata for FlatWorld FG(y), SixteenRooms, and FetchPlaceButton contain enabled epsilon transitions. The table below reports learned epsilon selection on FlatWorld FG(y), whose objective is to eventually remain in yellow.

Learned epsilon selection on FlatWorld FG(y). We use three training seeds, 1M physical steps per seed, H=50H=50, and 100 stochastic evaluation episodes per seed. Entries are means across the three seed-level results; continuation is measured conditional on reaching acceptance.

MetricMean across seeds
Episodes reaching acceptance100.00%
Conditional continuation through HH100.00%
Accepting physical visits per episode47.95

All three seeds learn epsilon commitment followed by sustained accepting behavior through HH. The mean of 47.95 accepting physical visits per episode reflects continued execution in the environment, since epsilon transitions do not contribute physical accepting-visit rewards. These results demonstrate learned automaton commitment together with continuous control and continued acceptance over the evaluation horizon.

Reviewer_eWUJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents RACE, a reinforcement learning method for LTL objectives with Büchi acceptance in continuous control. It trains SAC on the product of the environment and a limit-deterministic Büchi automaton with two rewards. The first comes from stage discriminators adapted from TiLoIL (Fan and Zhu, NeurIPS 2025) but trained on the agent's own rollouts instead of demonstrations. A stage is a strongly connected component of the automaton states from which acceptance is reachable, computed after the edges leaving accepting states are cut. Each non-accepting stage has its own discriminator head, which separates segments that progress past that stage from segments that stall or fail. The second reward is the automaton potential of DRL$^2$ (Bagatella et al., TMLR 2025) with added stage rewards. As in TiLoIL, accepting states receive a fixed critic target. The analysis treats a single discriminator under idealized sampling and extends it to pooled and staged training. It shows that a finite-rollout reachability surrogate, scaled by 1−γT1-\gamma^T, lower-bounds an infinite-time reachability objective. Maximizers of that objective, when they exist, approach optimal Büchi satisfaction under uniform moment bounds as L→∞L\to\infty and γ↑1\gamma\uparrow1 with (1−γ)(L+1)→0(1-\gamma)(L+1)\to0. The experiments compare RACE with LCER, DRL$^2$ and CyclER on eleven navigation, manipulation and locomotion tasks by eventual-discounted return and report learning improvements on several of them.

Critical strengths

\textbf{RACE learns staged rewards without demonstrations.} The TiLoIL paper names dependence on demonstrations as its main limitation and, as future work, proposes combining its exploitation mechanism with exploration that uses the automaton's structure. RACE takes up that direction, removing the demonstrations and pairing stage discriminators with automaton-guided exploration. In Fig.~4 its plotted returns are positive on recurrent tasks where the three RL baselines stay near zero, namely Point Zone, FetchPlace Button and SixteenRooms. The most direct evidence that RACE's additions make demonstration-free stage learning work is in Fig.~12, where TiLoIL without demonstrations stays at zero on five tasks on which RACE's returns are positive (Carlo, Cheetah Flip, FlatWorld Cycle, Point Zone and Point Button). If the evaluation held up, a demonstration-free method for these tasks would be useful, and this is the main consideration in the paper's favor.

Critical weaknesses

\textbf{1. The analysis does not establish recurrence for the trained method, and its surrogate and critic target can favor failure over it.} The theory is correct, but Theorem 1 covers a single discriminator trained on same-policy data with suffix-reachability labels. The heads that feed Eq.~9 instead learn from segment-level labels and, as the paper notes, give localized progress signals rather than reachability estimates. For the trained heads, the staged extension shows only that each head estimates the label probability of its own training data, which the paper notes does not in general lower-bound Theorem 1's objective. The paper also notes that neither the labels nor the weight β\beta guarantee its staged recurrence condition. The surrogate depends on a rollout only through its last accepting time τ\tau, since JH=E[(1−γmin⁡(τ,T−1)+1)/(1−γT)]J_H=\mathbb{E}[(1-\gamma^{\min(\tau,T-1)+1})/(1-\gamma^T)]. A failing rollout with one accepting visit at its end therefore scores 11, while a recurrent cycle can score less (0.9440.944 with T=55T=55 and γ=0.99\gamma=0.99). The fixed accepting-state target credits completion ``independently of subsequent-round task returns'', so for a fixed reward the critic ignores all task return after the next accepting visit. With TiLoIL's value 1/(1−γ)1/(1-\gamma), it prefers accepting sooner and then failing over accepting later and cycling forever whenever ρ+γαH<1\rho+\gamma\alpha\mathcal{H}<1 (per-step reward ρ\rho, common policy entropy H\mathcal{H}), so for any ρ<1\rho<1 without entropy. Theorem 3, which the paper offers only as motivation, needs (1−γ)(L+1)M(1-\gamma)(L+1)M small for some L≥TL\ge T, where Theorem 1 connects it to JHJ_H. At the only reported discount (0.990.99) and the shortest rollouts (5050 steps), this term exceeds 11 once a failing policy in the class averages two accepting visits.

\textbf{2. The case for a small increment over TiLoIL rests on an evaluation that does not support it.} TiLoIL's stage discriminators already learn from the agent's own segments alongside demonstrations, and RACE's Eq.~9 has the form of TiLoIL's Eq.~8. RACE drops the demonstrations, groups stages into components and adds the DRL$^2$ potential with stage rewards, so its case rests on the experiments. The paper claims ``complex LTL objectives,'' but every main benchmark combines only reaching, ordered goals, recurrence and global avoidance, and apart from SixteenRooms uses two to four propositions. None uses Until or a conditional response, although the paper defines Until, and these are where LTL specifications are meant to have the clearest advantage over hand-written rewards. TiLoIL is absent from Fig.~4, and the comparison in Fig.~12 does not support the claim that RACE consistently outperforms TiLoIL with demonstrations. At matched steps TiLoIL leads on Carlo until its curve stops at 400k steps and ties RACE on Doggo Navigate. On Fetch Avoid it runs 50-step episodes against RACE's 100 and levels off near 3737, close to the cap of about 4040 for 50 steps. Fig.~4 itself cannot be read as reported, since its returns exceed what the only reported discount allows (Question 1).

Questions for authors

\begin{enumerate} \item (Clarification.) Which discount and episode lengths produce the eventual-discounted returns in Fig.~4? The main body states neither. A return near 370370 on Doggo Avoid needs a discount above 0.9970.997 and more than 370370 accepting visits. Under the only discount the paper reports (0.990.99), its stated 50-step and 100-step Fetch horizons cap returns at about 4040 and 6464, yet Fetch Avoid shows returns near 9090. Relatedly, the Point-task formulas omit G\mathrm{G} before the safety terms, so as written they constrain only the first step, and the two formulas appear attached to each other's task descriptions. Which formulas were used in the experiments? \item (Clarification.) For a fixed reward, the accepting-state target removes all task return after the next accepting visit from the critic. How does RACE avoid preferring acceptances that lead to failure? In reach-avoid tasks such as Fetch Avoid, the unsafe region must still be avoided after the target is reached, yet the fixed target is the same for every action at an accepting state. What signal other than entropy tells the actor to keep avoiding it? \item (Clarification and requested experiment.) Beyond removing demonstrations and grouping states into components, how do RACE's stage labels and reward differ from TiLoIL's? Please add TiLoIL with demonstrations to Fig.~4 at matched step budgets and episode lengths, together with TiLoIL with demonstrations plus RACE's automaton potential. Only the exploration term differs between these two, so the pair tests the claim that the gains come from exploration. \item (Requested experiment.) On the current benchmarks, how does RACE compare with a hand-designed staged reward (stage index, distance to the next region, and a penalty for unsafe regions) under the same SAC setup? \item (Requested experiment.) RACE needs one head per non-accepting stage, yet its high-dimensional tasks have at most three stages, the ten-goal scaling study uses a two-dimensional point agent, and the shared-head alternative, described as only slightly worse, stays near zero on Fetch Align (Fig.~6). How does RACE perform on a high-dimensional robot whose specification has eight or more non-accepting stages, or Until and conditional-response obligations? \end{enumerate}

2 comments

Thank you for the questions.

Finite-rollout guarantees

Late acceptance can give failures high scores; the guarantee needs explicit assumptions.

C.4, Corollary 1, under A1 and B1–B2′, gives for πH∗∈arg⁡max⁡π∈ΠJH(π)\pi_H^*\in\arg\max_{\pi\in\Pi}J_H(\pi):

B∗−B(πH∗)≤Rlog⁡(T+1)T+CfailZT.B^*-B(\pi_H^*)\le \frac{R\log(T+1)}{T}+\frac{C_{\mathrm{fail}}}{Z_T}.

BB is satisfaction probability, B∗=sup⁡π∈ΠB(π)B^*=\sup_{\pi\in\Pi}B(\pi), and ZT=∑t=0T−1γtZ_T=\sum_{t=0}^{T-1}\gamma^t. Uniformly over the policy class, RR bounds satisfying-trajectory waiting times; CfailC_{\mathrm{fail}} bounds expected last accepting time plus one on failures. The terms vanish as T→∞T\to\infty and γ↑1\gamma\uparrow1. These constants are uniform over the fixed policy class and independent of T,γT,\gamma.

Under B1–B2, geometric weighting sharpens the count bound:

B^*-B(\pi_{L,\gamma}^*) \le\frac{R}{L+1} +\min\!\left\{(1-\gamma)(L+1)M,\;1-\gamma^{(L+1)M}\right\}. }$$ $M$ bounds expected accepting visits conditional on failure. At $\gamma=0.99$, $L=50$, $M=2$, the failure term improves from $1.02$ to $1-0.99^{102}\approx0.6413$; the full bound $R/51+0.6413$ may remain loose. Choosing $\gamma=1-(L+1)^{-2}$ gives gap $(R+M)/(L+1)$. B2′ gives $R/(L+1)+(1-\gamma)C_{\mathrm{fail}}$. These conditional results do not certify the trained policies. ### Staged rewards and progress For balanced, pooled segment labels, C.5 Proposition 4 gives: $$\left|\widehat J_{k,g}^{\mathrm{st}}(\pi)-J_{H,g}^{\mathrm{st}}(\pi)\right| \le K_g\sum_C\rho_C(\pi)\lambda_C^{\max} \left(\sqrt{\eta_{k,C}/2}+2\delta_{k,C}(\pi)\right).$$ $\eta_{k,C}$ is fitting error; $\delta_{k,C}$ is replay/label/balancing mismatch. We have not established that these errors are small. Since $q_h^\pi(x)\le q_{C,h}^\pi(x)$, progress need not imply acceptance. Proposition 5 requires liveness and advancement probability at least $p$ within $\ell$ steps, yielding $p^m$ acceptance probability within $m\ell$ for rank $m$. Recurrence requires this after acceptance too; neither labels nor $\beta$ ensures it. ### Actions after acceptance D.1/Algorithm 1 uses $C=1/(1-\gamma)$ and structural failure set $F_\bot$: $$\begin{aligned} \widehat y&=R(x,x')+\gamma\widetilde V(x'),\\ \widetilde V(z)&= \begin{cases} C, & z\in F,\\ 0, & z\in F_\bot,\\ \mathbb E_{a\sim\pi(\cdot\mid z)}[Q_{\bar\theta}(z,a)-\alpha\log\pi(a\mid z)], & \text{otherwise}. \end{cases} \end{aligned}$$ Completion earns arrival credit. Outgoing actions depend on rewards and successors; failure retains immediate reward but no bootstrap. This provides feedback beyond entropy, not a safety guarantee. The uniform $Q(x,a)=C$ boundary simplifies theory. D.2 gives the entropy-free comparison; D.3 derives $\rho+\gamma\alpha\mathcal H<1$ for early segment completion. D.4 relates outgoing values to future acceptance under its exact construction. Recurrence still requires C.5’s conditions. ### Extended-horizon behavior We separately report accepting visits and subsequent violations: |Environment (H steps)|H|5H|10H|Post-acceptance violation: H/5H/10H| |-|-:|-:|-:|-| |Carlo (500)|2.40±1.27|11.83±14.59|20.67±32.06|25.00%/75.00%/75.00%| |Cheetah Flip (1,000)|14.63±9.47|21.34±21.49|21.34±21.49|N/A| |FlatWorld Cycle (50)|3.33±1.49|5.50±9.95|11.25±20.35|0.00%/0.00%/0.00%| |SixteenRooms (1,000)|38.67±33.49|195.33±169.16|391.33±338.90|0.00%/0.00%/0.00%| |PointZone (1,000)|2.20±2.05|12.00±11.02|24.00±22.05|0.00%/0.00%/0.00%| |FetchPlaceButton (500)|1.97±2.39|6.98±12.33|13.23±24.82|N/A| Reported seeds: 4/3/10/3/5/4 in row order. Frozen deterministic policies use 100 episodes/seed, except PointZone: one; no learning or acceptance resets. Entries are mean ± sample SD across seed means. Violations count accepting episodes whose first violation follows acceptance, averaged equally over eligible seeds. N/A: no safety clause. Carlo’s visits increase while violations rise from 25.00% to 75.00%. Continued progress is not safe recurrence. Visits need not be distinct physical completions; zero conditional violations is not an infinite-horizon guarantee. ### Diagnostics with defined progress AUROC Final policies/heads are frozen: two seeds, 100 fresh stochastic episodes/seed at H. Each head scores its own nonaccepting component; labels concern strictly subsequent progress/acceptance. AUROC measures ranking; ECE calibration. Ranges span component–seed pairs; parentheses count defined AUROCs. “One class” means undefined. |Environment|Progress AUROC ↑|Acceptance AUROC ↑|Progress ECE ↓| |-|-:|-:|-:| |Doggo Navigate|0.734–0.986 (4)|0.734–0.904 (4)|0.068–0.209| |Doggo Avoid|0.949 (1)|0.949 (1)|0.293–0.397| |Carlo|0.910 (1)|One class|<0.001 to 0.363| Doggo Navigate replicates ranking; Doggo Avoid and Carlo add progress-ranking evidence. This does not establish universal calibration or the theorem’s assumptions.

Return metric and horizons

Figure 4 shows raw acceptance returns (undiscounted, bounded by H); γ=0.99\gamma=0.99 applies to training. Doggo uses 500-step episodes; Fetch uses 100. The caption, main text and P.2 clarify this.

TiLoIL Fetch runs replace H=50H=50 with H=100H=100, using RACE’s raw metric:

TaskOnline stepsRACE referenceTiLoILTiLoIL + potential
Fetch Avoid400K91.79±1.7776.31±5.4361.49±18.48
Fetch Align2M47.82±35.0733.40±35.5136.87±35.08

Point-task formulas

EnvironmentFormula used
PointZoneGF(b & XF g & XF r) & G!hazard
PointButtonGF(button2 & XF button0) & G!gremlin

Appendix H corrects swapped names/missing G. Parallel XF conjuncts do not impose green-before-red order.

TiLoIL and potential

RACE labels downstream segments positive, failed/current/upstream segments negative, excluding incomparable branches Appendix (G). Stage-aware exploration supports learning without demos.

30 controlled runs: five tasks × two demonstration-based TiLoIL variants × three seeds. Demonstrations, settings, horizons, budgets and evaluation match; only potential differs. Please refer to Reviewer nEJ4, “Baseline protocol.”

EnvironmentOnline stepsRACE referenceTiLoILTiLoIL_wo_demoTiLoIL_w_potential
Fetch Avoid400K91.79±1.7776.31±5.4386.91±0.4161.49±18.48
Fetch Align2M47.82±35.0733.40±35.5137.91±38.3936.87±35.08
Carlo1M2.55±0.623.88±1.170.00±0.004.18±0.57
Cheetah Flip1M14.50±8.6222.74±2.070.00±0.0012.07±9.20
FlatWorld Cycle2M3.64±1.491.33±1.890.00±0.000.00±0.00

Potential helps TiLoIL on Carlo/Fetch Align, not uniformly. Figure 7 tests RACE’s stage conditioning and potential.

Hand-designed rewards

Stage progress and distance replace discriminators:

\begin{cases} -1, & \text{on rejection or a declared safety violation},\\ +1, & \text{in accepting/completed states without failure},\\ \dfrac{3k+\exp(-d/L)}{3N}, & \text{otherwise}, \end{cases}$$ $k,N,d,L$ denote completed/total stages, next-goal distance and geometry scale. Failure takes priority; counts restart each round. No demonstrations/discriminators. Three seeds, 100 deterministic episodes/seed at H; eventual discount $\gamma=0.99$, unlike Figure 4; mean ± sample SD across seed means. |Task|Hand reward|Hand + potential|RACE (matched evaluation)| |-|-:|-:|-:| |Carlo|1.04±0.72|2.25±0.83|2.37±1.62| |Fetch Avoid|39.27±34.01|58.98±0.24|59.01±0.21| |Fetch Align|19.98±34.60|0.00±0.00|33.25±29.88| |Cheetah|1.03±0.73|4.53±4.72|13.65±9.47| |FlatWorld|1.31±2.27|3.94±0.00|2.96±1.97| |Doggo Avoid|43.08±38.75|39.91±46.27|95.31±4.78| |Doggo Navigate|9.83±9.39|9.39±9.18|44.22±38.30| |Task|Budget|H|L|Selection rule| |-|-:|-:|-:|-| |Carlo|1M|500|42.5 m|Waypoint-center separation| |Fetch Avoid|400K|100|0.19815 m|Initial gripper x-distance to goal boundary| |Fetch Align|2M|100|0.1 m|Target cube displacement| |Cheetah|1M|1,000|π rad|Maximum shortest angular distance| |FlatWorld|2M|50|1.58114|Maximum goal-center separation| |Doggo Avoid|400K|500|1.4 m|Reset-to-goal-boundary distance| |Doggo Navigate|400K|500|1.4 m|Consecutive goal-center separation| Fixed coefficients/scales; shared SAC: two 256-unit layers, twin critics, learning rate `3e-4`, batch 64, replay `1e6`, discount 0.99, target update `1e-4`, and automatic entropy tuning. Violation rates average per-seed unsafe-episode fractions. Potential improves four tasks; Carlo/Fetch Avoid violations fall from 29.00%/33.33% to 0.00%/0.00%. This does not prove discriminators necessary. ### Temporal tasks and scaling We test continuous 2D FlatWorld’s original/small-hazard layouts: - U-GR: `(!r U g) & F(r) & G(!b)`. - U-YG: `(!g U y) & F(g) & G(!b)`. - CR-YR: `GF(y) & G(y -> (!g U r)) & G(!b)`. CR-YR requires repeated yellow-triggered red-before-green responses, avoiding blue. Hazard-free layouts test six/eight-goal sequences; Carlo adds two waypoints for `0→2→1→3`. **Mean across three seeds:** 1M physical steps/seed, 100 stochastic episodes/seed. |Environment and task|$H$|Metric|Mean across seeds| |-|-:|-|-:| |Small-hazard FlatWorld: U-GR|50|Until success|100.00%| |Small-hazard FlatWorld: CR-YR|50|Conditional-response success|100.00%| |Original FlatWorld: U-YG|50|Until success|66.67%| |FlatWorld: sequence `1→2→3→4→5→6`|50|Sequence success|100.00%| |FlatWorld: sequence `1→2→3→4→7→6→5→8`|50|Sequence success|33.33%| |Carlo: repeat `0→2→1→3`, avoid crashes|500|Physical-cycle completion|83.33%| CR success requires at least two responses without safety/ordering violations through H; pending obligations are unresolved, not failures. Carlo’s physical cycles are not automaton acceptance; crash rate: 13.00%. For FG(y), refer to Reviewer mtfm, “Continuous controls and discrete automaton actions.” Figure 6’s shared head “slightly worse” understates the loss on Fetch Align. Separate heads matter on Fetch Align, this supports the importance of separate component-wise heads on Fetch Align
Reviewer_nEJ4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes RACE, a reinforcement learning method for LTL objectives represented using limit-deterministic Büchi automata (LDBAs). The method learns dense rewards from the agent's own rollout data. Specifically, it trains component-wise discriminators to provide feedback on progress through the automaton and combines these learned rewards with automaton-based potential shaping to guide policy learning.

Critical strengths
  1. The paper evaluates the proposed method on a broad range of tasks, including navigation, robotic manipulation, and locomotion, with several tasks involving recurrent LTL specifications.
  2. The theoretical appendix is detailed and explicit about the limits of its guarantees.
Critical weaknesses
  1. The contribution is incremental relative to prior work. The stage-wise discriminators and accepting-state Q-value target follow Fan & Zhu (2025), and the automaton-level potential shaping follows Bagatella et al. (2024).
  2. The theory does not analyze the algorithm that is implemented. Theorem 1 characterizes a single acceptance discriminator trained on on-policy data with discounted time sampling, and Theorems 2–3 concern the corresponding reachability objective. The implemented method instead uses component-wise discriminators with the staged reward in Eq. (9), whole-segment labels, and class-balanced sampling from replay data.
  3. The main evaluation metric, finite-rollout eventual-discounted return, is a proxy that cannot certify infinite-horizon satisfaction as the authors acknowledge. In addition, four of the eleven main tasks are non-recurrent. The extended-horizon evaluation in Appendix also covers only four tasks, at extended horizon. As a result, the experiments provide limited evidence on the infinite-horizon acceptance problem that motivates the paper.
  4. The paper contains several inconsistencies that raise questions about the reliability of the reported experimental details. For example, Appendix states that continuous-control runs take about 3 hours, while Table 1 reports substantially longer training times for many tasks, sometimes exceeding 50 hours.
  5. The main text is so dense that the paper’s central claims are hard to identify. Moving some secondary theoretical explanations to the appendix and using the available space to provide more details about the experimental setup would improve readability and make the empirical results easier to assess.
Questions for authors
  1. How were the baseline methods tuned? Were all methods evaluated using the same episode horizons, environment-step budgets?
  2. How does the fixed accepting-state Q-value boundary allow the policy to distinguish between safe and unsafe actions immediately after an accepting visit, given that the critic assigns the same target value to all actions at accepting states?
  3. Can the authors provide additional evaluations that distinguish successful accepting rounds from subsequent safety violations, particularly over longer execution horizons?
2 comments

Thank you for the questions.

Contribution

RACE couples TiLoIL-inspired component rewards with DRL²-inspired exploration, without demonstrations. Components handle nonaccepting cycles and incomparable branches; rewards capture intermediate progress and within-stage physical differences (Eqs. (5), (9); Appendix G). Figure 7 supports both components.

Baseline protocol

Figure 4 matches online budgets. LCER adds counterfactual replay, DRL² potential, CycLER cycle relabeling. Fixed SAC: two 256-unit layers, twin critics, learning rate 3×10−43\times10^{-4}, batch 64, replay 10610^6, training γ=0.99\gamma=0.99, automatic entropy tuning.

Baseline settings. Each cell reports episode horizon H / warm-up steps.

TaskLCERDRL²CycLER
Fetch Avoid100/2K100/2K100/2K
Fetch Align100/2K100/2K100/2K
Carlo500/4K500/4K1,000/10K
Cheetah1,000/2K1,000/2K1,000/2K
FlatWorld50/20K50/20K50/10K

The 30 new runs comprise five tasks × two TiLoIL variants × three seeds; only potential differs. Training/evaluation H=100/100/500/1,000/50 in table order. Demos contain 250/255/2,489/500 observations from five trajectories, and 5,000 from 100 FlatWorld trajectories. New Fetch runs replace Figure 12's H=50 with H=100. RACE uses four seeds for Fetch Avoid/Carlo/Cheetah and three for Fetch Align.

TiLoIL comparisons across five benchmarks. Each of the two TiLoIL variants uses three training seeds and demonstrations; RACE uses no demonstrations. Entries are mean ± SD.

EnvironmentOnline stepsRACE referenceTiLoILTiLoIL + potential
Fetch Avoid400K91.79±1.7776.31±5.4361.49±18.48
Fetch Align2M47.82±35.0733.40±35.5136.87±35.08
Carlo1M2.55±0.623.88±1.174.18±0.57
Cheetah Flip1M14.50±8.6222.74±2.0712.07±9.20
FlatWorld Cycle2M3.64±1.491.33±1.890.00±0.00

Returns are raw and undiscounted.

Extended-horizon acceptance and safety

Accepting visits at the original and extended horizons. Accepting-visit entries are mean ± SD.

Environment (H steps)H5H10HPost-acceptance violation:H/5H/10H
Carlo (500)2.40±1.2711.83±14.5920.67±32.0625.00%/75.00%/75.00%
Cheetah Flip (1,000)14.63±9.4721.34±21.4921.34±21.49N/A
FlatWorld Cycle (50)3.33±1.495.50±9.9511.25±20.350.00%/0.00%/0.00%
SixteenRooms (1,000)38.67±33.49195.33±169.16391.33±338.900.00%/0.00%/0.00%
PointZone (1,000)2.20±2.0512.00±11.0224.00±22.050.00%/0.00%/0.00%
FetchPlaceButton (500)1.97±2.396.98±12.3313.23±24.82N/A

Seed counts are 4/3/10/3/5/4 in row order. Frozen deterministic policies use 100 episodes/seed, except PointZone's one trajectory/seed, without learning or acceptance resets. Rollouts supply H/5H/10H prefixes. Sample SD is across seed means. Conditional violations count accepting episodes whose first violation follows acceptance, averaged equally over eligible seeds; N/A means no safety clause.

Continued progress and later safety failures are distinct; finite tests do not certify infinite-horizon satisfaction.

Theory and implementation

Theorem 1 analyzes an idealized discriminator. Appendix C.5: Proposition 4 separates fitting error from replay/label mismatch. Proposition 5 requires liveness and advancement probability ≥p within ℓ steps, yielding acceptance probability ≥p^m within mℓ steps for rank m. Recurrence requires this after acceptance too.

Final frozen-policy/head tests use two seeds and 100 fresh stochastic episodes/seed at H. Heads score their own nonaccepting components; labels concern subsequent progress/acceptance.

Final-checkpoint diagnostics for tasks with defined progress AUROC. Ranges cover visited component–seed pairs; parentheses give the number with defined AUROC. “One class” denotes undefined AUROC.

EnvironmentProgress AUROC ↑Acceptance AUROC ↑Progress ECE ↓
Doggo Navigate0.734–0.986 (4)0.734–0.904 (4)0.068–0.209
Doggo Avoid0.949 (1)0.949 (1)0.293–0.397
Carlo0.910 (1)One class<0.001 to 0.363

Doggo Navigate replicates ranking; Doggo Avoid and Carlo add progress-ranking evidence.

Safe action selection after acceptance

Appendix D.1/Algorithm 1 uses C=1/(1−γ)C=1/(1-\gamma) and

\widetilde V(z)=\begin{cases} C,&z\in F,\\ 0,&z\in F_\bot,\\ \mathbb E_{a\sim\pi(\cdot\mid z)}[Q_{\bar\theta}(z,a)-\alpha\log\pi(a\mid z)],&\text{otherwise}. \end{cases}$$ Completion receives arrival credit; outgoing actions depend on rewards and successors. Failure retains immediate reward but no bootstrap. The uniform $Q(x,a)=C$ boundary simplifies theory. ### Reporting and clarity We corrected PointZone’s budget to 4M in Appendix I, Table 1. Appendix P.4 will clarify timings: Fetch Avoid, 4.76 hours/400K; Fetch Align, 30.99 hours/2M, including timing scope and hardware. We will move derivations to the appendix and clarify assumptions/settings beside the results.

For Table TiLoIL comparisons across five benchmarks, the table compares RACE with TiLoIL and TiLoIL augmented with RACE's automaton potential. The two demonstration-based TiLoIL variants form a controlled 30-run study: five benchmarks, two variants, and three training seeds. Within each task, they share the environment, specification, demonstrations, warm-up, optimizer, tuning settings, horizon, online budget, and evaluation protocol; only the potential differs. Demos contain 250/255/2,489/500 observations from five trajectories, and 5,000 from 100 FlatWorld trajectories. All three methods are reported at the listed online interaction budgets, extending the Carlo comparison to 1M steps. The new TiLoIL Fetch runs use H=100H=100, matching RACE's Fetch horizon.

Comments on the paper

1 comment

Thank you for the detailed responses and new experiments. The corrected Fig. 4 metric, the matched Fetch Avoid runs, the hand-designed reward comparison and the Until and conditional-response tasks resolve several of my concerns. Two critical issues remain. Eq. 21 in the submission assigns C to every action at an accepting state, while the response describes a different target, so please clarify which one produced the reported results. Your matched runs also show TiLoIL with demonstrations ahead of RACE on Carlo and Cheetah Flip, so the Appendix N claim that RACE consistently outperforms it is not supported, although the results do support RACE being competitive without demonstrations. I am therefore keeping my score at 2 and will revisit it if the revision resolves the critic target and restates the Appendix N claim.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.