- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies backward Bellman updates on a rooted tree, where each node combines child values through a specified aggregation rule. It derives a matrix condition for contraction that accounts for both propagation through children and feedback through policies recovered from the previous value profile. A separate analysis characterizes aggregation changes that preserve local Gibbs policies and bounds the policy differences that can arise when aggregation sensitivity is reduced. The numerical experiments examine local decision changes, controlled feedback dynamics, and neural implementations. The guarantees apply to the stated node objectives. Interpreting the resulting policies in a game requires additional consistency conditions concerning rewards, transitions, and policy recovery.
Critical strengths
The stability analysis separates dependence on recovered ancestor policies from propagation through newly solved child values and derives explicit sensitivity bounds on a bounded invariant domain. The weighted certificate can establish convergence even when the update amplifies perturbations in the original value coordinates.
The decision-preservation analysis identifies which aggregation changes preserve every local Gibbs policy under the stated assumptions and quantifies possible policy differences when sensitivity is reduced. The assignment experiments evaluate each candidate under the original objective, and the matched controls distinguish continuation scaling from changes in route weighting. These comparisons show that normalization can change which actions are preferred.
Critical weaknesses
The theoretical advance beyond established results is limited. The matrix certificate combines standard contraction perturbation bounds with the tree dependencies and weighted-norm analysis. Theorem 4.12 uses the common-shift property of softmax, stated explicitly by Skalse et al. in Invariance in Policy Optimization and Partial Identifiability in Reward Learning, ICML 2023, Lemma C.5. Theorem 4.13 provides a direct bound on aggregation differences and a binary construction. The sharp scalar threshold concerns contraction in the ordinary sup norm, and its exact example remains contractive in a weighted norm beyond that threshold. Error amplification in the original coordinates still matters. The paper does not establish when a bound in the original coordinates is needed despite contraction in a weighted norm.
The application of the combined analysis remains underdeveloped. The assignment study concerns local choices, while the ancestor-feedback example in Appendix I.2 has identical local action values and uniform oracle policies. The neural results describe task-dependent behavior with incomplete retraining records and leave the required sensitivity bounds unverified. A branching example with nontrivial action choices and ancestor feedback would show whether the certificate predicts the behavior of the recovered policies. For a game interpretation, it should also identify the returns represented by the local future term and the aggregated child term in Equation (2), and verify the compatibility conditions in Appendix H.12. The examples treat these issues separately and do not demonstrate the combined result in a single decision problem.
Questions for authors
Please specify how the local future term and the aggregated child term in Equation (2) contribute to the return, and which sequential problems satisfy this decomposition. A supporting example should identify the policy recovery map and verify the conditions in Appendix H.12, including agreement between the policies used in the iteration and the local oracle policies. For which applications is it necessary to control errors in the original coordinates, as in Corollary 4.14? It would help to show how this requirement affects the choice of aggregation when convergence can already be established in a weighted norm.