- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies fixed-policy evaluation with fixed linear features. Starting from , checkpoint performs exact least-squares fitted Bellman updates on the same dataset; corresponds to the LSTD solution. The paper asks which checkpoint to retain when only that dataset is available.
On the theoretical side, the paper expands the iterates jointly in the reward, transition, and Gram-matrix errors. At order , the risk of a finite checkpoint contains a signed term, , which couples the checkpoint's population bias with the estimator's second-order mean shift. Variance-only calculations miss this term. The paper illustrates it with two examples and shows how treating the Gram matrix as fixed can introduce spurious noise.
The authors then propose a score that compares each candidate with the LSTD endpoint. The score includes a covariance correction and an endpoint-curvature correction, . Because the candidate's own curvature cancels, the same score applies to any smooth candidate family, including ridge. The theoretical results are:
- A calibration error of order for a fixed candidate (Theorem 18);
- The impossibility of exactly unbiased risk estimation from finitely many Bernoulli transitions (Theorem 17);
- An oracle inequality with an additive term of order (Theorem 2), along with a local version for ordinary iid transitions that uses condition-number gating and a fallback to zero (Theorem 4, Proposition 5);
- A proof that this additive term is at least eight times the endpoint risk (Proposition 7);
- For the single-state model, an exact characterization of when curvature improves selection for any and any finite candidate library (Theorem 8), an explicit example in which curvature helps (Proposition 6), and a proof that any advantage eventually vanishes (Corollary 9).
The experiments use four synthetic suites.
Critical strengths
-
Exact results and careful claims. The paper gives exact rational results that readers can verify, and it demonstrates its own limitations:
- Proposition 7 shows that the oracle bound cannot guarantee an improvement over LSTD.
- Section 5.4 reports that, at , the score bias is about twelve times the true risk.
- Appendix G reports that the conservative certificate fails on every dataset.
-
The cancellation identity and a matched experimental protocol. Eq. (71) shows that the score needs only the endpoint's curvature, so the same correction applies to ridge and to union candidate libraries. The authors use this property to make the comparisons fair:
- The same data budget for all methods;
- The same score across candidate families;
- Paired confidence intervals;
- Fixed baselines tuned on independent pilot data, which distinguish a weak candidate family from a weak selector.
Critical weaknesses
-
The experiments show that curvature helps very little, and its benefit cannot be predicted in advance.
- Curvature lowers the risk of the selected candidate by only 0.9 to 2.2 points, and the selector still trails a fixed choice tuned on pilot data by about 13 to 25 points.
- The frozen-Gram ablation outperforms the full score on the weighted (.51 vs. .57), sampling (.53 vs. .59), and iid (.55 vs. .57) suites, and ties it on the tabular suite. Yet joint Gram uncertainty is exactly what Section 3.3 and the contributions paragraph emphasize.
- With the same score, ridge outperforms checkpoint selection on all four suites.
- Proposition 11 relies on a construction with zero true variance, so it does not establish how often this effect matters.
Theorem 8 shows that, in the scalar model with a positive reward, the correction always favors smaller predictions and therefore induces more shrinkage toward the zero initialization. The observed gains could thus reflect either better calibration or extra shrinkage in weak-signal settings. Because the paper reports neither the selected and nor any strong-signal suite, it cannot distinguish between these explanations.
This matters because checkpoint selection for exact linear FQE has limited practical appeal on its own: both the endpoint and ridge are cheap to compute, and the paper claims no computational savings. The empirical case therefore rests entirely on a small effect that may be an artifact of the chosen experimental settings.
-
Beyond the scalar model, the theory does not answer the paper's own question.
- The selection guarantees are too loose. The additive terms in Theorems 2 and 4 are at least eight times the endpoint risk, so neither theorem can establish a gain over LSTD.
- The calibration constants break down at the sample sizes studied. In Section 5.4, the score bias is 0.105, whereas the risk is only 0.0085.
- Theorem 8 depends on the unknown true value . It sums the identity over the counts, so it identifies when curvature helps only if is known.
- The sample sizes required by Corollary 9 are astronomical. For the paper's own example (, , candidate library ), its sufficient condition requires . For the experimental candidate library (, ), the same formula gives about at and about at .
- The multidimensional case offers only diagnostics. Beyond the scalar model, the paper provides only a test that can rule out a switch in the selected candidate.
The tools are all standard: the second-order delta method, SURE-style corrections, bounded differences, and constrained risk lower bounds. The scope is also narrow: fixed features, , 24 states, realizable rewards, zero initialization, and iid transitions.
Questions for authors
- Please report the distributions of the selected and under the full score, the score without curvature, and the frozen-Gram score. Please also add a strong-signal suite in which the endpoint is nearly optimal, as well as a suite with misspecified features. If the curvature gains persist in these settings, that would support the calibration explanation; if they reverse, that would indicate they come from extra shrinkage.
- Did LSTD-Tournament, ridge with a holdout set, and the fixed eight-update method use the same gating and fallback to zero as the score-based methods? If not, how were the 891 datasets that triggered the fallback counted in each method's normalized risk? How would the iid results change if the thresholds were chosen without population margins, for example from independent pilot data?
- Why does the frozen-Gram score still win on the grouped suites when the population covariance is used?
5 comments
Response to Reviewer qiCE (2/3): selected K and λ, new suites and the frozen Gram (W1, Q1, Q3)
Q1: selected and (Appendix U.1, Tables 3–4). On gate-passing datasets, the full score's most common choices are , the zero prediction (31–38% of datasets), and , the largest ridge value (28–38%), against 30–34% and 26–36% for the no-curvature score. Most of the full score's checkpoint choices have . Curvature changes the selected checkpoint on 12–23% of datasets, and the full score wins on 55–63% of these switches. The frozen-Gram and full scores coincide in the tabular suite. In the other suites, the frozen-Gram score picks an earlier checkpoint than the full score on 25–49% of datasets and a later one on about 1% or fewer.
Calibration does not improve where the gains are (Appendix U.3, Tables 6–7). For , the full score minus true risk averages −.160, −.088, −.051 and −.103 times the endpoint risk in the four suites, while the no-curvature average is near zero. At the smallest the full score's absolute error is also larger, for example .40 against .18 in the tabular suite at (Table 6), the setting with the largest gain (−5.22 points, Table 7). At the largest sizes the gains are small or absent: −0.29 [−0.63, 0.05] in tabular at and +0.02 [−0.21, 0.25] in iid at .
Q1: higher signal (Appendix U.4, Table 8). We drew new iid datasets at signal levels 1 and 2 instead of .06 and .35, and the best pilot-chosen checkpoint then has .76–.77 of the endpoint's risk. Curvature no longer helps, and at level 2 it slightly hurts, which by your criterion points to shrinkage: full minus no curvature is +0.13 [−0.35, 0.60] at level 1 and +0.65 [0.16, 1.15] at level 2. At , where that checkpoint has .93 of the endpoint's risk, the difference is +0.52 [0.12, 0.92].
Q1: misspecified features (Appendix U.4). We added to the value, with the original signal level. It is orthogonal to every feature, because the features are a constant and cosines and sines of with , where . The gain persists, at −1.12 [−1.37, −0.88] against the projected fixed point and −1.10 [−1.31, −0.88] against the true value. By your criterion this points toward calibration. The suite is still weak-signal, however, since the best pilot-chosen checkpoint has .43 and .48 of the endpoint's risk, and the pilot mean of again removes the gain.
A larger design (Appendix U.5, Table 9). We drew new datasets with 48 states, and , gating each configuration at half its population margins. Full minus no curvature is −0.73 [−0.87, −0.59] overall, −1.71 at and −0.04 [−0.16, 0.09] at . Here the 1.5-fold penalty is worse than the full score, by 0.74 [0.31, 1.17].
As you say, the benefit cannot be predicted in advance (Appendix U.6). Within each original suite, configurations with a larger median ratio gain more, with Spearman correlations between this median and full minus no curvature from −.56 to −.77. However, the median cannot be computed from one dataset, mainly tracks and does not transfer across designs.
Q3: why freezing still wins in the grouped suites (Lemma 23, Appendix U.7). Prop. 11 only shows existence. The grouped suites (weighted and sampling) use features , where is the group-indicator matrix and a fixed orthogonal matrix. Every estimator then depends only on and , built from the group transition frequencies and mean group rewards , and not on the weighted group masses . Gram fluctuations therefore never move any estimator. The frozen-Gram derivatives along changes in are generally nonzero, however, so the frozen score adds variance that no estimator has. The population covariance cannot remove it, because the error lies in the derivative. Empirically, the frozen score also underestimates the risk of early candidates, so it selects earlier checkpoints and larger ridge parameters and shrinks more (Tables 3, 4 and 6). This extra shrinkage wins where shrinkage pays, but not consistently at higher signal: at signal level 2, full minus frozen is +2.81 [0.13, 5.50] in the weighted suite but −6.01 [−9.34, −2.69] in the sampling suite (Table 11). Lemma 23 does not apply to the iid suite, where the population covariance reverses the frozen-Gram advantage at the original signal levels (Table 14). At signal levels 1 and 2 the frozen-Gram score loses to the full score there by 37.63 and 59.17 points (Table 8). Sec. 1 now says that joint treatment is needed for a correct risk approximation but not necessarily for better selection.
Response to Reviewer qiCE (3/3): effect size, ridge, theory and gating (W1, W2, Q2)
Effect size and the pilot gap (W1). We agree that the gains are modest: 0.89–2.20 points (Table 2). The pilot-tuned fixed baselines estimate , the comparator in Thms 2 and 4, from 1,000 independent pilot datasets per setting, and, as you note, Table 1's 13–25-point gaps show that our selector is far from it.
Ridge and practical appeal (W1). We agree that choosing for exact linear fitted Q-evaluation is not a compelling target on its own and that ridge is the better family under our score. We claim no computational saving. Ridge selected by our score beats holdout-and-refit ridge in every suite, by 17.46–24.66 points, and checkpoint selection by the full score beats LSTD-Tournament in three suites and ties it in the weighted suite (Table 13). Shrinking the endpoint to , with no selection, is about as good as score-selected ridge in iid. Against checkpoint selection by the full score, it gains 3.31 points and ridge gains 3.49 (Tables 9 and 13). This shrunken endpoint is better than score-selected ridge in our larger design (46.7 against 48.6, Appendix U.5) but worse in the tabular and grouped suites (Tables 2, 5 and 13).
Oracle bounds (W2). Prop. 7 proves that the additive term of Thm 2, which Thm 4 shares, is at least eight times the leading endpoint risk, so neither bound can certify a gain over LSTD. Sec. 1 now presents the oracle bound only as keeping adaptive same-data selection at the endpoint's order- risk.
Calibration at the studied sizes (W2). We agree; our own experiments are affected too. In the cyclic iid configurations with and , the population ratio is (29/1600)/(1/8) = .145 (Appendix G). These configurations are thus worse conditioned than the example of Sec. 5.4, whose one-state value is .335. At , 23.4% of iid datasets fail the gate, and part 2/3 reports the realized calibration.
Thm 8 depends on (W2). Its proof is your identity summed over the counts, so Thm 8 maps the parameter space rather than giving a data-driven test, as Sec. 1 now says. For the Prop. 6 example, the exact risk of the no-curvature selector minus that of the full selector is +.087, +.030 and −.009 at , 16 and 32 (Appendix U.9, Table 15). At the full selector (.134) beats the endpoint (.273) but loses to the fixed checkpoint (.091).
Cor. 9 (W2). Your arithmetic is correct, and Cor. 9 is only an asymptotic sufficient condition. In your example the exact risks show that curvature stops helping at and that from on both selectors match the endpoint's risk to four decimals, so reflects slack in the condition. For the experimental bank, part 2/3 shows that the empirical gain is already small or absent at our largest sample sizes.
Scope and tools (W2). We agree that beyond one state we cannot say when curvature helps or hurts, and Sec. 1 now names this as the main open problem. The tools are standard, and one addition is the second-order cancellation: comparing each candidate with the endpoint cancels the candidate's own mean shift, so only the endpoint's shift must be estimated (Eq. 71). The new experiments relax zero initialization, realizability, and the 24-state design, while fixed features and iid blocks remain limitations (Sec. 6).
Q2: gating. Yes. In the iid suite the final output of every method goes through the same full-data gate, including the fixed eight-update method, the pilot-tuned baselines, and the refitted tournament and holdout-ridge outputs (Sec. 5.1, Appendix T.5). All methods therefore return zero on the same 891 failing datasets, which stay in every average with the loss of the zero prediction, so every paired contrast is exactly zero there. On the 6,309 passing datasets the ordering of the nine iid methods in Table 12 is unchanged, and full minus no curvature is −1.38 [−1.69, −1.07]. As you suspected, the primary thresholds came from known design margins, and Sec. 4.2 now says so. We therefore also set each test dataset's thresholds to half the empirical margins of its own independent pilot dataset of size . Fallbacks then drop from 891 to 228 for and to 585 for . The ordering is unchanged, and full minus no curvature is −1.20 [−1.47, −0.94] for and −1.39 [−1.63, −1.15] for . Sec. 5.4 now summarizes this.
If the revision addresses your concerns, we would be grateful if you would consider updating your assessment.
Response to Reviewer qiCE (1/3): calibration or shrinkage? (W1, Q1)
Thank you for a detailed and accurate review. Your shrinkage reading is correct. The curvature gain comes from data-adaptive shrinkage of the endpoint toward zero rather than from better calibration of individual candidates, and the revised abstract, Sec. 1 and Sec. 5.3 now say so.
What the curvature term does. Completing the square shows that the full score ranks candidates by their squared M-distance to the target t=h^−β^, plus the covariance penalty, and Sec. 4 now states this. In one state with known reward, β^=V∞/h^, where V∞ is the endpoint's estimated variance, so t=h^(1−V∞/h^2) is a variance-adaptive shrinkage of the endpoint toward zero, and Thm 8 says exactly when this shrinkage helps (Sec. 4.3). In any dimension, when the full and no-curvature scores select different predictions (a switch), the full score's choice f always has the smaller value of f⊤Mβ^ (Appendix U.1).
Controls (Appendix U.2, Table 5). Each control except the last replaces β^ in the curvature term 2(fK−h^)⊤Mβ^ by another vector, which we call a tilt, and keeps everything else. The table reports each score minus the no-curvature score.
The average of β^ over independent pilot datasets keeps the average tilt but not its variation, and it removes the gain. A random direction of the same norm also removes it, while flipping the sign hurts, so the gain needs a data-dependent tilt with the sign of β^. Larger multiples of β^ help more, and 3β^, our largest, is the best multiple in all four suites. A 1.5-fold penalty without curvature matches the full score in iid and beats it in the other three suites. We chose these multipliers on the test data and do not recommend them. Both are worse than no curvature in the higher-signal iid suite of part 2/3 (Table 8).
The shrinkage is toward zero, not toward the starting point (Appendix U.5, Table 9). With θ0=0, as in the paper, the two coincide. To separate them, we reran the iid suite on the same datasets with checkpoints started at 2θ⋆ or θ⋆/2, where θ⋆ is the true parameter, which no selector sees. If the gain were shrinkage toward the start, it would vanish from 2θ⋆, but full minus no curvature is −1.70 [−1.92, −1.48] from 2θ⋆ and −1.65 [−1.90, −1.41] from θ⋆/2. From 2θ⋆, curvature switches no longer favor earlier checkpoints (303 later, 250 earlier), and the full score wins on 82% of them. The 1.5-fold penalty, which shrinks toward the start, is worse than the full score by 2.12 [1.49, 2.76]. This matches the algebra, because the target h^−β^ does not involve θ0.
In theory, the correction still removes the leading-order bias of a fixed candidate's score (Thm 18).