- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes RACE for reinforcement learning under infinite-horizon LTL specifications, using a product of the environment and a limit-deterministic Buchi automaton. It supports continuous environment states and actions, with discrete automaton choices where enabled. Component-wise classifiers turn acceptance and partial progress in agent-generated trajectories into dense rewards, combined with automaton-based exploration potentials and SAC policy updates. The theory establishes a finite-rollout reachability lower bound and conditional links between an idealized objective and recurrent acceptance, rather than optimality of the learned SAC policy. Experiments compare RACE with LTL baselines across navigation, manipulation, and locomotion, including ablations and extended-horizon execution tests.
Critical strengths
The method extracts learning signals from partially successful trajectories without demonstrations. The experiments cover several domains, include component ablations and extended-horizon accepting-round counts, and report gains over relevant LTL baselines. The analysis explicitly distinguishes its conditional surrogate results from guarantees for the implemented learner.
Critical weaknesses
The evaluation protocol needs reconciliation. Appendix H specifies 50-step Fetch episodes, while Figure 12 reports horizons of 100 for RACE and 50 for TiLoIL. The evaluation discount or scaling also needs clarification: Figure 4 contains returns above 100, whereas the unit-reward metric in Appendix P.2 would be bounded by 100 if it used the reported training discount of 0.99. These may be reporting differences, but they currently limit interpretation of the comparisons.
Equation 21 also assigns the same Q target to every action at accepting states. In the ideal critic limit, Equation 22 then supplies only an entropy incentive at those states. How safe, recurrent continuation is learned under this boundary rule remains unclear.
Questions for authors
- Continuation after acceptance. Equation 21 gives every action at an accepting state the same target C. In the ideal critic limit, Equation 22 therefore leaves only an entropy incentive there. How does the actor learn to choose safe actions that begin subsequent rounds? Safety-violation rates and successful continuation rates would help interpret the accepting-round counts in Appendix M.
- Connection between theory and implementation. The conditional optimality result concerns a global optimizer of an idealized objective, while the practical heads use balanced, pooled whole-segment labels (Appendices C.5 and G). Could a calibration check against current-policy future-progress probabilities, or a matched-suffix-label ablation, show how closely the practical rewards reflect the reachability motivation?
- Continuous and discrete actions. How are enabled discrete automaton epsilon-actions selected alongside continuous SAC controls? Do any evaluated tasks require these choices, and what policy architecture or selection rule handles them?
2 comments
Connection between theory and implementation
We directly test whether practical discriminator scores predict subsequent progress and acceptance under the current policy. At the final training checkpoint, we freeze the policy and discriminators and collect 100 fresh stochastic episodes per seed at horizon for each reported task, using two training seeds. Each discriminator is evaluated within its own nonaccepting component, with labels based on strictly subsequent outcomes. We assess ranking with AUROC and calibration with expected calibration error (ECE).
These diagnostics provide replicated evidence in Doggo Navigate: both component-wise discriminators give informative progress and acceptance rankings in both training seeds. Doggo Avoid and Carlo provide additional progress-ranking evidence. The table below summarizes these diagnostics.
Final-checkpoint diagnostics for tasks with defined progress AUROC. Ranges cover visited component–seed pairs; parentheses give the number with defined AUROC. “One class” denotes undefined AUROC.
| Environment | Progress AUROC ↑ | Acceptance AUROC ↑ | Progress ECE ↓ |
|---|---|---|---|
| Doggo Navigate | 0.734–0.986 (4) | 0.734–0.904 (4) | 0.068–0.209 |
| Doggo Avoid | 0.949 (1) | 0.949 (1) | 0.293–0.397 |
| Carlo | 0.910 (1) | One class | <0.001 to 0.363 |
These measurements complement Proposition 4 in Appendix C.5, which characterizes the discrepancy between the learned reward objective and current-policy progress, accounting for fitting error and distribution mismatch. Figure 7's component ablations provide complementary evidence for the contribution of discriminator feedback to learning performance.
Evaluation protocol and reported returns
Figure 4 reports raw acceptance returns: the undiscounted sum of unit acceptance rewards over an episode. The reported is the SAC training discount; the plotted raw return is bounded by the episode horizon . This explains returns above 100 in Doggo's 500-step episodes and returns near 90 in RACE's 100-step Fetch episodes. We have corrected the Figure 4 caption, the main results description, and Appendix P.2 to state the evaluation metric explicitly.
Our new TiLoIL and TiLoIL-plus-potential Fetch runs both use , matching RACE, at matched online interaction budgets. This replaces the TiLoIL setting in the original Figure 12 comparison. The two TiLoIL variants use the same demonstrations and tuning settings; RACE uses no demonstrations. Both TiLoIL variants use the same raw acceptance-return definition as RACE. The results are included below:
Fetch comparison at and matched online budgets.
| Task | Online steps | RACE reference | TiLoIL | TiLoIL + potential |
|---|---|---|---|---|
| Fetch Avoid | 400K | 91.79 ± 1.77 | 76.31 ± 5.43 | 61.49 ± 18.48 |
| Fetch Align | 2M | 47.82 ± 35.07 | 33.40 ± 35.51 | 36.87 ± 35.08 |
We have updated Appendix H and the Figure 12 description to state the horizons for the original and new comparisons explicitly.
Continuous controls and discrete automaton actions
Our implementation uses a learned hybrid policy to select between physical control and automaton epsilon transitions. A masked categorical selector chooses either the physical-action branch or an enabled epsilon edge. On the physical branch, a tanh-Gaussian policy supplies the continuous action. The selector and continuous controller are trained jointly using action values and entropy terms, with invalid epsilon choices masked. The policy therefore learns both when to commit and which enabled transition to take.
An epsilon action changes only the automaton state. It does not move the simulator, advance physical time, consume a new physical label, or receive a physical accepting-visit reward. Subsequent physical actions are evaluated under the new automaton state.
The compiled automata for FlatWorld FG(y), SixteenRooms, and FetchPlaceButton contain enabled epsilon transitions. The table below reports learned epsilon selection on FlatWorld FG(y), whose objective is to eventually remain in yellow.
Learned epsilon selection on FlatWorld FG(y). We use three training seeds, 1M physical steps per seed, , and 100 stochastic evaluation episodes per seed. Entries are means across the three seed-level results; continuation is measured conditional on reaching acceptance.
| Metric | Mean across seeds |
|---|---|
| Episodes reaching acceptance | 100.00% |
| Conditional continuation through | 100.00% |
| Accepting physical visits per episode | 47.95 |
All three seeds learn epsilon commitment followed by sustained accepting behavior through . The mean of 47.95 accepting physical visits per episode reflects continued execution in the environment, since epsilon transitions do not contribute physical accepting-visit rewards. These results demonstrate learned automaton commitment together with continuous control and continued acceptance over the evaluation horizon.
Thank you for the questions on post-acceptance control and discriminator validation. We address each below.
Continuation after acceptance
Response. Our implementation uses the action-dependent target in Appendix D.1 and Algorithm 1. The constant C supplies completion credit on arrival at acceptance; actions taken from accepting states are evaluated through their immediate rewards and successor states. The actor therefore receives task-dependent action values in addition to the entropy incentive. The uniform boundary Q(x,a)=C simplifies the theoretical analysis.
With C=1/(1−γ) and structural failure set F⊥, the implemented target is
\widehat y&=R(x,x')+\gamma\widetilde V(x'),\\ \widetilde V(z)&= \begin{cases} C, & z\in F,\\ 0, & z\in F_\bot,\\ \mathbb E_{a\sim\pi(\cdot\mid z)}[Q_{\bar\theta}(z,a)-\alpha\log\pi(a\mid z)], & \text{otherwise}. \end{cases} \end{aligned}$$ Every action, including one selected at acceptance, is evaluated through its own reward and successor state. Entering structural failure retains the immediate reward but receives no future bootstrap; continuing toward another accepting visit retains the corresponding future credit. D.2 gives the explicit task-value comparison, which applies without entropy. In D.4’s exact, horizon-consistent reachability construction, outgoing action values are proportional to the probability of another accepting visit. The critic therefore provides action-dependent feedback for the next segment. We evaluate policies using this implemented target at $H$, $5H$, and $10H$ without further training or resets at acceptance, reporting accepting visits and post-acceptance violations separately. **Accepting visits at the original and extended horizons.** Accepting-visit entries are mean ± SD. | Environment (H steps) | H | 5H | 10H | Post-acceptance violation: H / 5H / 10H | |---|---:|---:|---:|---| | Carlo (500) | 2.40 ± 1.27 | 11.83 ± 14.59 | 20.67 ± 32.06 | 25.00% / 75.00% / 75.00% | | Cheetah Flip (1,000) | 14.63 ± 9.47 | 21.34 ± 21.49 | 21.34 ± 21.49 | N/A | | FlatWorld Cycle (50) | 3.33 ± 1.49 | 5.50 ± 9.95 | 11.25 ± 20.35 | 0.00% / 0.00% / 0.00% | | SixteenRooms (1,000) | 38.67 ± 33.49 | 195.33 ± 169.16 | 391.33 ± 338.90 | 0.00% / 0.00% / 0.00% | | PointZone (1,000) | 2.20 ± 2.05 | 12.00 ± 11.02 | 24.00 ± 22.05 | 0.00% / 0.00% / 0.00% | | FetchPlaceButton (500) | 1.97 ± 2.39 | 6.98 ± 12.33 | 13.23 ± 24.82 | N/A | Seed counts are 4/3/10/3/5/4, for Carlo/ Cheetah/ FlatWorld/ SixteenRooms/ PointZone/ FetchPlaceButton, respectively. Frozen deterministic policies run without learning or resets at acceptance, with 100 episodes per seed; except PointZone, which uses one deterministic trajectory per seed. Each rollout supplies the H/5H/10H prefixes. Visits are reported as mean ± sample SD across seed means. Post-acceptance violation rates measure the fraction of accepting episodes whose first violation follows first acceptance, averaged equally across seeds. N/A indicates no safety clause. Mean accepting visits increase beyond the original horizon in all six recurrent benchmarks, with approximately proportional growth in SixteenRooms and PointZone. These tasks and FlatWorld Cycle show no observed post-acceptance violations. Cheetah Flip achieves approximately 46% more accepting visits at $5H$ under demanding six-degree-of-freedom flip dynamics. Carlo achieves 8.6 times as many accepting visits at $10H$, with further growth from $5H$ to $10H$ while the observed post-acceptance violation rate remains unchanged. These results provide direct evidence of continued execution without additional training.