- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper provides a highly-constrained SoftMax-Transformer setup that aims to learn a pre-constructed TD-operator from finite noisy supervision(via an attention matrix W to form a weighted combination of the classical Bellman residuals). Only three TD coefficients and n(n-1) attention scores are trained with the remaining architecture frozen. On deployment, the model parameters are frozen, policy evaluation and greedy policy improvement iteratively take place on an single trajectory collected by a fixed behavior policy. Policy performance bounds and a high-probability suboptimality certificate are established under the stated assumptions.
Critical strengths
- The paper is fluent with proper introduction and related work citations, and uses an additional table to help reader understand what are the incremental contributions. Formulas and algorithms are beautifully organized that greatly enhances readability.
- A clear theoretical flow from Theorem A.2 and Corollary A.3 eventually landed in an connection between the finite noisy operator training and the returned policy quality. A high-probability suboptimality guarantee is proved by explicitly analyzing four types or errors.
- The paper explicitly addressed the potential statistical dependence derived from the same reused deployment trajectory. Adaptive policy selection and repeated evaluation on single trajectory are enabled by a simultaneous concentration bound over deterministic target policies.
Critical weaknesses
- The paper provides an additional finite-sample and finite-step operator-error guarantee w.r.t. the work of Xie et al. (2026). However, those might heavily rely on the carefully designed identification probes, training curriculum and the highly-constrained Transformer architecture. Though the authors acknowledges those limitations, an additional baseline using ordinary MDP trajectories to jointly train TD coefficients and attention scores would be favorable for comparison.
- The theoretical guarantee that a single-reused-trajectory could support repeated policy evaluation is heavily dependent on the strong full state-action coverage and the geometric mixing assumptions. Though limitation acknowledged, certain state-action pairs might have zero stationary probability under a fixed behavior policy, causing the assumption to fail while very low stationary probabilities might require an impractically large sample budget.
Questions for authors
- To what extent do the probes and curriculum affect the training procedure? Will this method generalize to ordinary MDP trajectories or mixed-data joint training that matches the provided budget?
- Could the authors further investigate how a behavior policy could potentially satisfy Assumption 3.1 in practical deployment or whether the conditions should be relaxed?
- The authors acknowledge that the sufficient certificate bounds remain uninformative given the tested budgets, therefore, could the authors provide an example of a nontrivial certificate (C_J < 2B) that is computed without privileged true-MDP information?
2 comments
Thank you for the detailed response and revisions. Proposition A.13 and the new results in Appendix C.1.7 meaningfully address my concerns about coverage-dependent certification and the absence of a nontrivial certificate example. I appreciate your clarification of the distinction between the record-based certificate and the coverage-dependent a priori guarantee, as well as the clearer comparison with Xie et al.
Concern about training generality remains. Proposition B.10 explains why a global joint-GD guarantee does not follow from the stated excitation conditions alone, but does not directly replace the requested ordinary-trajectory joint-training comparison, which is still deferred to the final version. The training guarantee remains bounded to designed probes and the constrained architecture. The reported nontrivial certificates are also limited to checkpoints trained with noiseless labels and evaluated on small tabular tasks.
I appreciate these substantive improvements and will maintain my overall rating.
Thank you for the precise reading of the contribution relative to Xie et al. The revised PDF marks all additions in blue.
Q1 / W1 (role of probes; ordinary trajectories?). Singletons make the three coefficients identifiable independently of attention; two-record reward probes make each free score identifiable through r0crℓ(Axu). The runs use a mixture of 3/4 ordinary examples and 1/4 probes, so probes need only be present at mass pmin. On ordinary-trajectory joint training we have a proved obstruction rather than an untested gap: Proposition B.10 gives a noiseless label law with every probe at positive mass and teacher excitation, yet a nonoptimal stationary point of the joint loss. Hence the staged guarantee, with local joint GD as an optional refinement. We agree your baseline is the right empirical complement and will add it to the final version, reported as succeeding without a guarantee if that is what we find.
Q2 / W2 (coverage and mixing). Your concern is correct: the sufficient occupation threshold is conservative and grows as μ0 shrinks, so small stationary probabilities do translate into large budgets for the advance guarantee.
We chose to remove the dependence rather than defend it. New Proposition A.13 shows the deployed certificate does not need Assumption 3.1: the finite-loop recursion holds on the residual event alone, with the realized teacher norm ρj=∥I−Wm+γTmπj∥∞ computed from the record, the deterministic operator bound and clipping, and nothing downstream reintroduces the occupation event. Assumption 3.1 now buys only an a priori rate. We measure ρj=0.756 on the ring and 0.765 on the random-kernel family, against certified 0.895 and 0.964 (Table 6). A pair with zero stationary probability is never observed, and no estimator can evaluate it; but the learner now detects this on its own record and reports 2B when a factor reaches one. We do not take the pessimistic-readout relaxation because the lower envelope is not computed inside the network; §3.1 now gives this as the rationale for the selection slack.
Q3 (non-trivial certificate without privileged information). Yes, for the trained model: the five frozen checkpoints give C8<2B in all 750 cells at m=1,048,576 (mean 5.595), and mean 3.009 against 6.667 at m=4,194,304, realized gap 0 (Appendix C.1.7). Its inputs are the realized norm (record and policy table), the residual radius (record and known constants), and LG∥θ−θ⋆∥2, a deterministic function of the checkpoint. None uses P or r; the true model only reports the realized gap. Two caveats: the primary radius requires deterministic rewards (our banks have them), while the mixture radius, which needs only bounded rewards, crosses at 2,097,152 (mean 5.868); and the certificate is informative, not tight.
Xie et al. Table 1 now separates the premises, which makes the trade explicit: their softmax result is for prescribed parameters, with asymptotic loss optimality and an on-policy trajectory of the evaluated policy; ours trains by finite noisy GD and re-encodes one behavior trajectory for each new target policy, at the price of designed probes and a constrained architecture.