- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Speculative Rendering: a method motivated from speculate decoding in LLMs, to pre-render likely future simulation states before the driving policy/planner has executed that far. The rationale for speculative rendering is to reduce idle GPU time, by proactively rendering multiple likely-futures states in parallel as a batch, rather than the slower vanilla process of sequentially iterating between inferring policy actions and rendering the next simulation state. The trick is to understand whether the future renderings are “close enough” to be valid, or should they be discarded and when should the simulator indeed need a “true” rendering at some future time step.
While I really appreciate how well this problem was motivated, I have concerns about how much value the proposed solution provides. I've also left some questions too.
Critical strengths
- Well motivated: sensor simulation for closed-loop evaluation and/or fine-tuning of end-to-end autonomous stacks can be expensive and slow; especially when there’s communication delays between the stack/policy and the renderer that iteratively wait for each other before stepping, leading to inefficient use of compute. Paper seeks to reduce the GPU’s idle time and increase the parallelization of compute.
- Well written: the language was clear and the points logically presented. Figures are also clear, such as Figure 1’s simplicity, which is nice.
- Novel solution: I had not seen something like this before.
Critical weaknesses
Large weaknesses:
- While motivated from speculative decoding (positive point), the analogy of a draft-and-target models here seems weak given their often non-overlapping support, i.e. the (deterministic) renderer from a slightly shifted/erroneous simulation states will often be such that there is no change of them drafting a render that the “target model” could have rendered (if I understand correctly), so any analogy of rejection sampling here seems difficult (unless perhaps the state predictions themselves were probabilistic?), i.e. it would always “reject” ?
- Sanctity check: it looks like in Table 1 we can reduce the GPU hours by half, without compromising too much on PDMS. How would this all compare to a simple baseline, such as: predict actions at 0.5s, render at 1s (already). Figure 8b already shows ~80% re-use at k=1, so this simple baseline might work almost just as well?
Small weaknesses:
- Why not have “Speculative Rendering” or “SpecRender” in the title? (preference for “Speculative Rendering”). Possibly too late for ICLR, but if possible, perhaps worth consideration.
- Use a consistent method name. Both “Speculative Rendering” and “SpecRender” are used frequently, interchangeably. Pick one?
- Line 090: presenting example benefits without associated costs like “latency decreases from 364 to 138 ms” and “cost decreases from 840 to 488 H100 GPU-hours” is a bit meaningless given the aforementioned inherent trade off mentioned on Line 086. Perhaps pair it with the modest performance decrease (71.33 versus 72.13 PDMS) so the readers can see the trade off of using where the speculative renderings which contained small errors. Or perhaps there is a single metric the author could show, something like model performance per wall clock time of training (which presumably is better than vanilla training, given the commute usage increase >> model performance drop), for a given threshold chosen.
- Line 198: For equation (4), do the authors mean “s_{t’}” or “s_’{t}” ?
- Line 204: “predicts N future poses”: can the authors please clarify what N is. Is this N possible ego states at next time step (time t) given prior observations, or is N time steps into the future, predicted from t? Edit: I see later on Line 215 “K<=N” the answer is: steps into the future from t.
- Line 206. Why “a” in (5) but “x” in (2), pick one convention?
- Line 206: Consider using subscript “n” instead of “j” as a subscript over “{1,..., N]”. This is also confusing in equation 6 where j is re-used for {1,…, K}. Perhaps consider not telling the reader about the N future poses, since the superfluous N-K state predictions don’t matter; instead mention K predicts to match K renderings, and change “j” to “k”?
- Line 220: notation of using “t” in both the subscript and superscript seems redundant. Also the “t+j” of (6) doesn’t match the “t,j” of (5), pick one convention and stick with it? If the authors like the superscript “(t)” convention, then the subscript can simply be “j”. Whichever chosen, is Figure 2 yet another disagreement by using \tau ?
- Line 665: use {} to preserve capitalization for names in bibtex, e.g. {Gaussians} and {Gaussian}
Questions for authors
- Is there a way a user could understand how to set thresholds in (7) to understand (in advance) a trade off of policy quality with rendering error?
- If rendering is expensive, but simulator state errors are still small (measured by equation 9), could we take advantage of (cheaper) depth-based reprojection rendering given (1) speculative/cached renderings and (2) knowledge of the simulation state error (still within threshold limits)?
- Line 256: “Checking reactive actors in addition to the ego constrains position errors in the surrounding traffic, but can reduce the acceptance rate”: is the rendering cost for other agents much less than the surrounding (Gaussian splatt) scene? If so, is there a potentially more complex variant of this method which can leave draft-render the scene, but post-hoc render other agents efficiently?
- Could errors accumulate over time in unbounded ways? I’m unsure, but thinking about equation (9) which verifies a bound on the simulator state errors between (1) plans conditioned from current time, (2) re-plans conditioned from future speculative renderings; but never against (3) re-plans conditioned on future real renderings, had we played the simulator out normally. While computing (3) would defeat the purpose of this paper, are there any situations where (1) and (2) could stronger agree over long time horizons (i.e. always below verification threshold), but would acclimate error with (3) (had it been computed) unbeknownst to the user?
2 comments
Response to Reviewer cPNR (2/2)
- Unbounded error accumulation
Thank you for raising the question of cumulative rollout drift. We agree that Eq. (9) controls local draft-position error but does not bound trajectory separation from standard rendering over arbitrarily long horizons. For example, at an intersection, a small difference between an accepted cached image and a fresh rendering of the realized state could change the policy’s decision to turn, after which subsequent drafts might pass verification along a different trajectory.
Because AutoVLA predictions are stochastic, evaluating such counterfactuals is challenging. Fig. 5 shows closely aligned trajectories and similar reward distributions, while Appendix A.4 intentionally presents failure cases in which small differences are amplified through replanning and reactive traffic. Error accumulation is therefore possible, but our focus is its practical effect on the RL training signal, which appears limited: training with SpecRender achieves PDMS under standard-rendering evaluation, compared with for standard-rendering training.
Response to Reviewer cPNR (1/2)
We thank Reviewer cPNR for their time and constructive feedback. We address the concerns below.
Speculative decoding analogy
We thank the reviewer for the comment. As stated in L17 and L79, our method is inspired by the drafting and verification components of speculative decoding and we do not claim its distribution preservation guarantees. As the reviewer notes, with a deterministic renderer, rejection sampling would reject a draft whenever it differs from the target rendering. Because our verifier allows small drifts, speculative observations may shift subsequent policy predictions, producing rollouts slightly off-policy relative to standard rendering. We assume that a strict verifier keeps these differences small enough to preserve the RL training signal, as supported by Tab. 1 and Fig. 5. We will clarify this distinction.
Exact rejection sampling would require the entire framework to support probabilistic ego and traffic states, which is uncommon in closed-loop simulation. It would therefore require probabilistic planners and traffic controllers producing trajectory distributions. Instead, SpecRender is designed as a plug-and-play technique compatible with off-the-shelf planners such as AutoVLA and VADv2 (see upcoming response to R-e1oT).
A simpler way to enable rejection sampling would be to treat point-state estimates as centers of Gaussian distributions, with a diagonal scale hyperparameter λ controlling support overlap. This would serve a similar role to the acceptance thresholds ϵpos and ϵact. We will consider this probabilistic extension in future work.
Baseline experiment
Following the reviewer’s suggestion, we evaluate a baseline that renders every 1s while replanning every 0.5s using the latest available observation. This halves the number of rendering cycles but retains the sequential dependency. At t=0.5 s, the planner reuses the observation from t=0 while the simulator waits for the next action. At t=1 s, the planner stalls while the simulator renders the new observation, which is reused at t=1.5 s. This baseline reduces simulation-step latency from 364.48 ms to 264.12 ms, compared with 138.91 ms for SpecRender. The k=1 reuse refers to verified observations rendered in advance for the next simulation step, which explains the further latency reduction.
Setting the threshold
We evaluate candidate thresholds on held out scenes by measuring 1) L2 distance between speculated and target poses 2) PDMS degradation with respect to non-speculated trajectories and 3) latency. We then select the L2 distance that best compromises between latency and PDMS degradation. We further validate the hyperparameters by evaluating the policy’s open-loop L2 error, which in our case was approximately 0.5m at 2 sec lookahead. We will add a description of these procedures in the Appendix.
Depth Reprojection
Thank you for this interesting suggestion. We could render and cache depth maps alongside RGB images, then resample and “correct” the RGB images based on the difference between drafted and realized camera poses through depth reprojection. This could increase draft acceptance and avoid unnecessary asynchronous re-renders.
However, it would be beneficial only if the overhead of depth rendering, caching, and resampling were lower than that of a fresh render. Moreover, even a slight change in viewpoint might reveal content hidden in the original render, requiring to integrate the full appearance via other cached frames or additional targeted rasterizations. Finally, camera reprojection would only fix errors in speculated ego state, but not in actor motion, that would require to be handled separately. The idea is sound and it opens several interesting research questions, and we regard it as a future potential extension of SpecRender.
Selective Actor Rendering
Indeed, 3DGS rasterization cost scales linearly with the number of input Gaussians, which is typically much higher for the background than for other agents. We could therefore distinguish rejections caused by incorrectly speculated ego states from those caused by incorrectly speculated traffic. These cases could then be handled separately, e.g. by depth-reprojecting the background (as above), or by re-rendering only dynamic agents. We will explore these further optimizations in future work.
Minor suggestions on notations and title
Thank you for these suggestions. We will integrate the changes in the final version.