- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This work introduces PluralGen, an inference framework that eliminates redundant model execution in Sequential Monte Carlo (SMC) LLM reasoning, by grouping particles at identical complete states and evaluating each distinct state once per decode step. using a multiplicity-aware density head (MDH) and grouped-state runtime, PluralGen achieves up to a 3.2× speedup, an 88.1% reduction in KV cache allocation, and 2.9× higher serving throughput compared to SGLang baselines.
Critical strengths
- novel execution-search decoupling. PluralGen eliminates hidden compute waste by removing duplicate Transformer passes, and vocabulary projections for identical particle states that persist after resampling.
- particle semantics with grouped representation. PluralGen maintains exact statistical equivalence with standard sequence-wise SMC without expanding execution rows.
- multiplicity-aware density head dynamically switches between dense and counted block projections, which scales vocabulary computation with distinct occupied states, rather than nominal particle counts.
Critical weaknesses
- performance limits under specific conditions. speedups diminish under low state concentration or small populations (), and are bounded by long prompt pre-fill times because PluralGen strictly optimizes the decoding phase.
- the proposed technique strictly optimizes the decoding phase. As demonstrated in the paper's sensitivity analysis, for tasks with long prompt pre-fills, the end-to-end latency gains degrade because the fixed pre-fill time bounds overall speedup (Amdahl’s law).
Questions for authors
- What is the precise control overhead of group management when distinct-state ratios are high (), and how does PluralGen perform under high-temperature settings with minimal particle collisions?
- How do dynamic state counts and batching mechanisms interact with communication overheads in multi-GPU or distributed serving environments?
3 comments
W1 / W2.
-
Reducing redundant decoding still gives clear end-to-end speedups with long prompts. Longer inputs increase pre-fill cost, while particle search also requires continued generation across multiple candidates. Prefix caching reuses prompt KV history, and PluralGen continues to share model execution for identical states after resampling. These savings accumulate during decoding. The table gives the breakdown for 1.5B ASMC at . Inputs of about 2K and 3K tokens yield 1.94× and 2.25× speedups, with most savings coming from subsequent decoding. Paired times are ordered as baseline / PluralGen. The first step includes pre-fill, head execution, and control work.
Input tokens First step (s) Subsequent decoding (s) Full request (s) Speedup 35–150 0.031 / 0.016 8.018 / 6.115 8.089 / 6.153 1.31× 2,050–2,197 0.110 / 0.054 11.279 / 5.808 11.426 / 5.882 1.94× 3,074–3,149 0.141 / 0.082 9.031 / 3.999 9.218 / 4.094 2.25× -
Pre-fill optimizations can be combined with PluralGen. For a fixed decoding workload, a larger shared pre-fill cost makes the saved decoding time a smaller fraction of total latency. This is the relationship described by Eq. (48) in Appendix B.2. Pre-fill optimization and shared decoding act on different phases of a request and can improve end-to-end latency together.
Q2.
-
PluralGen retains clear speedups after accounting for communication and synchronization. Tensor-parallel ranks use the same grouped batch width, and graph buckets accommodate shape changes as state counts vary. Both systems use the same head, graph, and communication settings. End-to-end time is measured as the completion time of the slower rank. Results on MATH500 with TP=2 are below.
Model / Method Speedup at Speedup at 1.5B / ASMC 1.37× 1.95× 1.5B / Power-SMC 1.78× 2.10× 7B / ASMC 1.75× 2.16× 7B / Power-SMC 1.37× 1.47× -
Fewer execution rows also reduce the size of collective input tensors. For example, with 7B ASMC at , both decode execution rows and the cumulative AllReduce input volume computed from those rows fall by about 85.7%, including graph padding. Shared states thus reduce model computation and the corresponding collective inputs. Communication still has fixed launch and synchronization costs, so end-to-end speedup does not scale directly with the reduction in execution rows. The grouped batches nevertheless deliver the gains reported above.
Thanks author(s) for supplying additional results and the elaboration.
The paper's main strengths are its efficient reuse of identical particle states, preservation of SMC sampling behaviour, and substantial reductions in computation and KV-cache memory.
The additional experiments addresses my main concerns about high-temperature sampling, group-management overhead, long prompts, and multi-GPU execution. The results are encouraging, although performance still depends on how often particles share states, and broader distributed scalability remains less established. I appreciate these efforts and think the original score reflects its standing.
We appreciate the reviewer's questions about applicability, runtime overhead, and multi-GPU communication. And we have revised the paper in light of this feedback.
W1 / Q1.
PluralGen still delivers end-to-end speedups at high temperatures, even when reuse is minimal. ASMC and Power-SMC use sharpened proposals and resampling to allocate more computation to high-weight trajectories, so several particles continue from the same state. The baseline already shares inherited KV state. PluralGen also shares these particles' current model evaluation while preserving independent draws, which gives larger gains under the original search settings. Higher temperatures flatten the proposal and make particles diverge faster, reducing the computation that can be shared. The table compares the original settings with a proposal temperature of 1.75 on the 1.5B model, where ρ=U/N is the distinct-state ratio. Even when ρ reaches 98.1%, PluralGen retains a 1.08× speedup.
Group management has a control cost comparable to that of the sequence-wise baseline. PluralGen maintains offspring through member indices, counts, and KV references, avoiding copies of entire sequences and KV histories when multiplicities change. The table reports exclusive CPU time for Power-SMC at N=128, τ=2, and ρ=98.36%, directly comparing management costs when reuse is minimal. Including synchronization and control work common to both systems, the full control stage takes 7.908 ms/step for the baseline and 7.595 ms/step for PluralGen. These totals are measured separately from the exclusive component times below.
Serving throughput still improves substantially at small particle counts. When a request contains less redundant computation, its latency improvement is smaller, but the saved execution rows and KV space remain available to concurrent requests. In Table 2 of the submission, ASMC achieves 1.68–1.75× higher throughput at N=32 and concurrency 16. Scheduling by occupied states therefore still improves serving capacity when each request has a small particle budget.