- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies a spectral-target mismatch in Laplacian representation learning for non-reversible Markov chains. Forward geometric-lag sampling discounts the transition kernel before reversibilization, whereas preserving the Chung eigenvectors requires reversing this order. DALLO independently selects forward or time-reversed transitions at each substep and retains ALLO's augmented-Lagrangian optimization. Under stationary sampling and exact conditional transitions, the resulting estimator targets the intended energy. Experiments on discrete grid mazes, directed cycles, and coupled cycles report improved eigenvector recovery, including comparisons with one-step ALLO.
Critical strengths
-
A substantive diagnosis of sampling-induced target bias. The analysis separates population operator mismatch from optimization error and provides a principled sampler correction. The sufficient condition for eigenspace changes and the explicitly scoped local-stability result make the theoretical contribution more precise than a generic extension from undirected to directed graphs.
-
Informative experimental controls. Target-alignment experiments, one-step controls, three readout conventions, buffer-distribution audits, and repeated-seed evaluations directly examine the proposed mechanism. On uniform Cycle-240, the practical recovery scores are 0.9979 for DALLO, 0.4243 for discounted ALLO, and 0.9587 for one-step ALLO (Table 1). Appendix K also provides useful matched-horizon comparisons, with DALLO retaining substantial advantages in many settings.
Critical weaknesses
-
The empirical attribution remains incomplete, and the main text overstates the collection-horizon control. The primary comparison changes both the sampler and the reset horizon: 1,000 steps for DALLO versus 50 for ALLO. Contrary to Section 5's no-improvement statement, discounted ALLO on biased CycleNet-2 improves from 0.7856 to 0.9534 with longer trajectories (Table 27), exceeding DALLO's 0.6382 in the matched-horizon comparison (Table 21). Furthermore, independent buffers and method-specific density readouts change the empirical geometry. The grid-buffer audits show substantial departures from the stationarity assumed by the estimator, although DALLO's directed-buffer deviations are small. These issues do not invalidate the population theorem, but they limit attribution of the reported empirical differences to the sampler correction alone.
-
The practical significance remains insufficiently established beyond spectral recovery. All evaluations concern finite discrete graphs; there is no value-approximation, planning, or control result establishing when the corrected representation is preferable for an RL task. Incoming-record resampling also relies on repeated discrete states, while continuous-state use requires an additional reverse-conditional estimator. Its error and computational cost are not evaluated. The manuscript therefore establishes a useful operator-learning result, but provides limited evidence that its sampler offers a practical advantage in the broader RL setting motivating the work.
Questions for authors
-
Can DALLO and both ALLO controls be evaluated using a common transition corpus and matched collection horizons, with oracle and practical readouts? Please reconcile the biased CycleNet-2 result and revise the main-text summary. A matched comparison from existing buffers or logs that preserves the principal gains, together with a corrected account of the exceptions, would support reconsideration toward acceptance.
-
Which operator and weighting measure does the finite-buffer implementation actually estimate when the buffer is nonstationary? Please document incoming-transition sampling, resets, and endpoint handling, and quantify the resulting operator discrepancy on an existing difficult configuration. Small discrepancies accompanied by preserved recovery gains would address the gap between the estimator guarantee and implementation.
-
Which practical claims are intended beyond finite-state spectral recovery? Please distinguish established recovery benefits from prospective control benefits and state the reverse-sampling requirements explicitly. A precise, evidence-supported account of the method’s finite-state applicability would strengthen the case for acceptance.
4 comments
After posting our response, we ran two more experiments to check that the difference in collected data does not explain our results.
1. Same data for both methods on CycleNet-4 biased. We trained ALLO with and with on DALLO’s own buffer. All methods used DALLO’s buffer (1,000 chains of 1,000 steps per seed) for 10 seeds, and the same readout, the per-vector cosine against the Chung eigenvectors at the final checkpoint. ALLO used its published hyperparameters, and for the one-step version we changed only the discount.
| Method | Mean cosine |
|---|---|
| DALLO | 0.757 |
| One-step ALLO () | 0.516 |
| Discounted ALLO () | 0.382 |
DALLO beats both on all 10 seeds (paired -test, for each). With , moving ALLO from its own 1,000-step episodes to DALLO’s buffer takes it from 0.476 to 0.516 on the same 10 seeds (). In the CycleNet-2 shared-buffer run from our response, ALLO with goes from 0.954 to 0.934 (). So training ALLO on DALLO’s buffer did not change its results in a significant way, and on CycleNet-4 DALLO keeps its advantage with the same data and the same one-step target, which supports the proposed sampler correction. Because of the rebuttal time we ran 10 seeds and not the 30 of the paper, so these means have more seed noise. We will add a full shared-buffer evaluation of DALLO and ALLO to the revised paper.
2. Same data in the downstream task. In Appendix M the two methods were trained on different data. Released ALLO used its own 50-step episodes and , while DALLO used its own buffer, and we did not run one-step ALLO there. The aim of Appendix M is to show the downstream utility of DALLO’s one-step representation () relative to ALLO’s discounted representation (), and not to compare how well each method estimates the one-step operator. So Appendix M does not isolate the sampler correction. The CycleNet-4 experiment above and the directed problems (Table 1, Appendix K) do that. To check that the difference in data does not explain the downstream result, we trained ALLO with on DALLO’s buffer of the river lake (10 seeds), so both methods use the same data, and ran the same DQN. With this data ALLO gets 27.0%, lower than the 55.4% it gets with its own 50-step episodes, and DALLO (88.0%) wins on all 10 seeds. So the different data in Appendix M did not give DALLO an advantage.
Thanks for running the extra comparisons and fixing the discussion of biased CycleNet-2. Appendix P answers my question on the finite-buffer operator and clarifies how trajectory endpoints are handled. I also found the biased CycleNet-4 result helpful: on the same transitions, RALLO outperforms both versions of ALLO (including the one-step baseline), which addresses my concern about the collection protocol on that graph.
What I'm still unsure about is how far this carries over to control. My concern with the extensive rewrite is that the revised version puts more weight on the goal-reaching experiment to motivate the one-step target and demonstrate practical value. The river layout experiment does show a clear gain over discounted ALLO on the same data, but I still can't tell how much the proposed sampler itself adds over one-step ALLO for goal reaching, since that comparison was only run for spectral recovery. On the wider set of graphs, the 11/12 CycleNet result only matches the collection horizon, so biased CycleNet-4 is still the only setting where all three methods are trained on the exact same transitions.
Overall, the revision strengthens the finite-state recovery result and gives a concrete downstream example, but the evidence for the sampler's practical advantage is still too limited for me to raise my score. I'll keep my rating at 2.
Thank you for reviewing the additional experiments and for acknowledging that Appendix P resolves your questions about the finite-buffer operator and trajectory endpoints. We also appreciate your recognition that the CycleNet-4 comparison addresses the collection-protocol concern on that graph, and that the river experiment demonstrates a downstream gain on identical data.
We agree that a downstream comparison with one-step ALLO would further clarify how much the sampler’s improvement in spectral recovery translates into control. We also recognize the additional value of extending the identical-buffer comparisons across the benchmark suite.
We would, however, respectfully ask that the remaining empirical questions be weighed alongside the contribution already established. The paper provides an analysis of how discounting and reversibilization interact, together with a sampling construction that preserves the reversibilized one-step eigenspaces under discounting and the associated recovery guarantees. The controlled CycleNet-4 experiment demonstrates improved recovery over both ALLO variants, including the one-step method sharing the same target, when all three methods use identical transitions. The river experiment then provides a concrete case in which the resulting representations improve downstream control over discounted ALLO on the same data.
The purpose of the downstream experiment is to give practical support for the choice of target in a setting where one-step bottlenecks matter. We believe that this example, together with the operator analysis and controlled recovery result, provides a theoretically and empirically justified basis for the paper’s present claims.
Given the concerns that you now consider resolved and the results you acknowledge, we would be grateful if you would reconsider the overall assessment. If the breadth of the downstream evidence remains decisive, we would appreciate clarification of how that limitation is weighed against the theoretical contribution and the controlled empirical findings. This would help us and the area chair understand the remaining basis for your assessment.
Thank you again for the time and care you have devoted to the paper.
We appreciate the thoughtful feedback. We conducted the experiments you suggested and the results strengthen the case for the proposed approach.
We present the results for the shared buffer below. We will add these to the revised manuscript. For the downstream effectiveness of the proposed approach please see bullet 2. in the response to Reviewer z2jR or Appendix M in the revised paper.
We trained ALLO on DALLO's own replay buffers over biased CycleNet-2 (same 10 seeds, the same 106 transitions in 1000 trajectories of 1000 steps). It reaches 0.935 with the oracle and 0.933 with the practical readout, against DALLO's 0.637 and 0.638, so the gap is not caused by the data. With matched 1000-step collection horizons on all CycleNets (Appendix K, Tables 20 to 22), DALLO stays above both ALLO controls in 11 of the 12 settings, and biased CycleNet-2 is the only exception. This is a special case in which ALLO's sampling-induced target nearly coincides with the Chung eigenvectors (see the answer to Q3 of Reviewer tB56). We will revise the main-text summary and the account of this exception accordingly. Thank you for noticing this.
The implementation learns the operator that the buffer defines: I−21(P^π+Pπrev^) in L2(dbuf), where dbuf is the buffer's state distribution. We sample states uniformly from the buffer; at each substep, we choose a random occurrence of the current state and move forward or backward (the 0.1% of steps crossing an endpoint become self-transitions). This walk is reversible with stationary distribution dbuf. Thus, our unbiasedness and local stability results apply under the same assumptions, replacing dπ with dbuf. DALLO's stable equilibria are therefore the eigenvectors of this buffer operator.
When the buffer is stationary, this operator equals our target, as in the reset-chain view of Wu et al. (2019, Appendix B). In our directed settings, the buffer is nearly stationary (TV(dbuf,dπ)≤0.023), and the operator matches the exact eigenvectors at 0.991 per vector on CycleNet-2 biased. In contrast, on GridRoom-64 biased (TV=0.70), the buffer operator differs greatly from the target (0.24 even for a perfect learner), but DALLO recovers its subspace (overlap 0.98). Thus, the gap comes from the data, not the estimator. We will add this analysis to the appendix.
Our main claim is finite-state spectral recovery. On non-reversible chains DALLO learns the eigenvectors of I−Palt, which give the Chung Laplacian eigenvectors after scaling by D1/2. For control we now have evidence but no guarantee. In the new downstream experiment (See the bullet 2. in the response to Reviewer z2jR or Appendix M in the revised paper).
For the reverse sampling, the backward step itself only needs the predecessor of the current position in the stored trajectory. A jump to another occurrence of the same state is needed only when the walk changes direction, since going forward and then back on the same trajectory returns to the same state instead of giving a fresh sample. So the multi-step walk requires states that are revisited in the buffer, which holds in finite or discretized state spaces, and extending it to continuous states is future work.
If the reviewer would find it helpful in reconsidering their assessment, we would be happy to provide a preliminary revised manuscript incorporating the proposed changes, so that they can verify these changes before making their final recommendation.