- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Earlier work of Allen et al. (2024) introduced the so-called lambda-discrepancy as a one-sided diagnostic of inconsistent value prediction under insufficient memory in a POMDP. The measure is based on comparing two Q functions computed with multi-step temporal-difference learning of different trace parameters. The central issue that the paper addresses, is that in an off-policy setting, multi-step temporal-difference learning requires corrections, such as importance sampling, tree-backup, or retrace, all proposed in earlier works. The paper establishes that the lambda-discrepancy from Allen et. al. (2024) with retrace corrections and off-policy learning provides a one-sided diagnostic of inconsistent value prediction in POMDPs, thereby generalizing the result form Allen et. al. (2024) from on-policy to off policy.
Critical strengths
The writing is clear and the mathematical statements appear to be correct. Proofs have not been checked line by line, however.
Critical weaknesses
The algorithmic contributions are somewhat limited: lambda-discrepancy has been introduced before, the issue with multi-step temporal-difference learning in off-policy settings are known, and the retrace correction has also been introduced before (Munos, 2016). Nonetheless the detection theorem (Theorem 3) appears to be a nontrivial extension from the literature, e.g., due to the fact that there is a clipping operation.
My main concern is the connection between detection and memory learning. Nonzero exact discrepancy witnesses insufficient observations, but zero discrepancy does not certify sufficiency, and its magnitude need not rank memory representations by predictive or control quality. The authors acknowledge this limitation and illustrate nonmonotonicity in Figure 6. Consequently, the detection results alone do not establish that minimizing the proposed objective improves memory. The learning experiments provide empirical support, but further analysis would help explain when and why this objective is effective.
In particular, a systematic study varying history-window length or memory representation, and reporting lambda-discrepancy alongside prediction error and control return, would further clarify the relationship between the proposed diagnostic and a useful signal for memory learning.
Questions for authors
Can you isolate the contribution of the discrepancy loss? Please provide an ablation retaining both auxiliary retrace critics and their regression losses in (10) while disabling the discrepancy loss. This would help isolate the benefit of discrepancy minimization.
Can you extend Figure 6’s analysis to controlled memory representations or history-window lengths on the learning benchmarks? In particular the non-monotonicity of the discrepancy-measure is concerning.