- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how reinforcement learning can use polyhedral states directly. It formalizes states as finite unions of polytopes and pairs PPO or DQN with encoders for half-spaces, vertices, relational graphs, difference-bound matrices, and rasterized projections. The experiments compare concrete clock values with symbolic zones in timed automata, then compare encodings across five domains with native geometric structure. The results suggest that exposing geometry relevant to decisions can improve performance under a fixed training budget, although no representation is best across all tasks. I view this as a useful exploratory study of state representation. The motivation is reasonable, and both the positive and negative results are informative.
Critical strengths
- Many planning and control problems already provide feasible regions, clock constraints, and spatial adjacency relations, so using this structure directly is well motivated. The two levels of set aggregation also accommodate varying numbers of constraints or vertices and arbitrary component orderings.
- The experiments include informative contrasts. Geometric representations have a clear advantage on GridTimed, while the point baseline is competitive on ModeSel. The vertex encoder largely fails on TRMGrid, and there is no clear ranking on Rush Hour or Lawn Mower. These results help establish when the choice of representation matters.
- On PHA Navigation, H-Rep augmented with a few neighborhood statistics achieves the highest mean on the easy and hard tiers and exceeds the GNN on all three tiers. This is an interesting finding about the value of selected relational summaries compared with access to the full graph.
Critical weaknesses
-
Practical value and scope. I find the idea of giving RL direct access to geometry appealing, but would like a clearer account of which practical problems naturally provide the required polyhedral states and what prior knowledge is needed to construct and update them. The benchmarks cover several structured settings, but their connection to practical applications could be more concrete. One representative application could illustrate where the geometric input comes from, what difficulty existing approaches face, and how this framework helps. This would make its value and scope easier to assess.
-
Contribution relative to existing methods. The framework builds on established DeepSets, GNN, CNN, and RL methods, with its contribution lying mainly in the unified formalism, representation choices, and empirical insights. Please clarify what reusable capabilities or insights this perspective adds to existing approaches to structured state representation. The most direct evidence concerns gains over a point-based MLP under a fixed sample budget. For planning, control, or symbolic methods that already exploit geometric constraints, what difficulty is a learned policy intended to address?
Positioning the work primarily as a study of RL representations, and explaining how it complements domain-specific methods, would be reasonable. Broader claims of practical solving advantages would need corresponding evidence. My concern is the clarity of this positioning and motivation, rather than an exhaustive comparison with conventional algorithms.
-
Interpretation of the central comparison. The timed-automata experiments compare current clock values with zones, which describe sets of clock valuations satisfying constraints. Please briefly explain how these inputs correspond, what information the zones incorporate from the environment or execution history, and whether both representations preserve the same task and action semantics. This would help clarify the respective roles of structured encoding and supplied symbolic information in the observed gains.
Questions for authors
- Which practical problems are best suited to this framework? Could one representative application illustrate how polyhedral states are obtained and what limitation of existing approaches motivates their use?
- What is the main added value over existing structured RL representations and domain methods that directly handle constraints? Which aspects are supported by the current results?
- How do concrete states and zones correspond in the timed-automata experiments? What information does each input contain, and are they compared under the same task and action semantics?