acceptodds
Under review as a conference paper at ICLR 2027

LaneRoPE: Positional Encoding for Parallel Reasoning and Generation

Abstract

Parallel LLM test-time scaling techniques (e.g., best-of-) require drawing sequences conditioned on the same input prompt. These methods boost accuracy while exploiting the computational efficiency of batching generations. However, each sequence in the batch is traditionally generated independently and hence does not reuse intermediate generations, computations, or observations from other sequences. In this paper, we propose LaneRoPE to enable coordination and collaboration among sequences at generation time. LaneRoPE involves two key ideas: (a) an inter-sequence attention mask to make sampling of sequences dependent on one another; and (b) a RoPE extension that injects positional information that captures relative positions between tokens, both within and outside a particular sequence. We evaluate our approach on mathematical reasoning tasks and find promising results: LaneRoPE enables collaboration among sequences, yielding additional accuracy gains under limited generated sequence length. Importantly, since LaneRoPE enables coordination with minimal changes to the underlying LLM architecture and introduces a negligible overhead at inference time, it is appealing to rapidly incorporate parallel reasoning into existing LLM inference pipelines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
RoPE, parallel inference, LLM, reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_LDWROpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes LaneRoPE. LaneRoPE extends RoPE with an additional lane dimension to train models to use information from other reasoning paths. Experiments show improvements in majority-voting accuracy for the 7B model, with little additional inference latency.

Critical strengths

1.Simple and practical design. LaneRoPE extends positional encoding to both token and lane dimensions while preserving within-lane positional relationships. It can be integrated into pretrained models with minimal architectural changes.

2.Low inference overhead. The method introduces relatively few additional parameters and enables cross-lane information sharing with negligible inference latency overhead.

Critical weaknesses

1.The mechanism of cross-lane collaboration is not sufficiently analyzed. For the 7B model, moving from independent generation to four-lane collaboration barely changes the average individual accuracy, while majority-voting accuracy improves from 63.0 to 64.1.

2.The conditions under which collaboration is beneficial remain unclear. With the same KTO-trained LaneRoPE model, four-lane collaboration improves the 7B model's average maj@4 from 63.0 to 64.1 but reduces the 1.5B model's score from 52.4 to 51.3. These results suggest that the effectiveness of collaboration depends on model capacity and training strategy.

Questions for authors

1.When one lane provides correct evidence or incorrect reasoning, do other lanes use it to correct their reasoning or simply follow it? In particular, when only a minority of lanes are correct, does collaboration improve or hurt the final result?

2.Why does four-lane collaboration improve the 7B model but hurt the KTO-trained 1.5B model? Is the degradation caused by the additional context or interference from other lanes?

2 comments

We thank the reviewer for highlighting the simple and practical design of our method as well as its inference efficiency. Below we try to address the reviewer's main concerns.

  • W1 Can you provide further analysis of the collaboration mechanism? Why majority voting improves while individual accuracies remain stable? In the new analysis below, we find that LaneRoPE KTO reduces unanimous-correct and minority-correct solutions while increasing majority non-unanimous correct solutions: this explains the improved maj@4 scores despite minimal individual accuracy changes and suggests LaneRoPE's learnt behavior goes beyond simple lanes agreement.

Precisely, we compare independent generation (N=1) with joint generation (N=4) for the 7B LaneRoPE and Bridge models; see also Apx. D.4. For each set of 44 jointly generated answers to each question, the table below reports the fraction of them containing zero to four correct answers (averaged over the 4 maths benchmarks). Notably, Bridge KTO has an almost inverse trend with respect to LaneRoPE KTO, showing a clear increase in unanimous correct groups; improvements in maj@4 come mostly from reduced one-correct (minority correct) groups. Under SFT, LaneRoPE increases pass@4 (less zero-correct cases) but also unanimity; again, Bridge shows a mostly inverted trend, with more minority answers but also reduced pass@4.

ModelN0 correct1 correct (minority)2/3 correct (majority)4 correct (unanimous)
LaneRoPE KTO135.699.0313.0742.21
435.097.6617.3139.94
Bridge KTO137.679.5917.8734.87
437.618.9015.3838.11
LaneRoPE SFT140.6312.2221.2225.93
439.2412.2521.8426.68
Bridge SFT140.3810.4021.3327.88
441.8412.2820.9224.96
  • W2 Dependency on model capacity and training Of course, training data and model capacity are key factors to determine the resulting collaboration capabilities as our results confirm. We observe this is orthogonal to the method adopted, as Bridge suffers from the same problem. Nevertheless, LaneRoPE's benefits still appear in the SFT regime, where its expressive attention to the full history enables learning more useful policies from the same data.

  • Q1 With correct lanes minority, does collaboration improve or hurt results? Do lanes simply follow each other? Our new analysis in Apx. D.4 suggests collaboration can recover from correct minority and LaneRoPE doens't learn to simply follow majority. We compare the full distributions of correct-answer counts under independent (N=1N=1) and joint (N=4N=4) generation on matched problems, with both LaneRoPE and Bridge. Under KTO, when four independent answers contain only one correct answer, joint generation produces more correct answers on average. While SFT models tend more toward agreement, LaneRoPE's joint generation more frequently produces correct answers where independent generation produced none, contributing to its higher pass@4 rate. See also W1.

  • Q2 Why does cross-lanes collaboration hurt the 1.5B model with KTO? We suspect this to be a combination of model capacity and the model-dependent data filtering in KTO (we retain only problems with at least a solution and no more that 50% correct responses), because of which the 1.5B dataset includes typically simpler problems than the 7B-generated one. In this regime, we suspect learning a stronger independent strategy might be simpler than learning to use cross-lanes information. We will further investigate this aspect.

Thank you for the additional analysis. It helps explain the changes in answer distributions, although the collaboration mechanism and the degradation in the smaller model remain insufficiently understood. I will maintain my original rating of Weak Accept.

Reviewer_N5V5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper combines cross-lane causal attention with a lane-aware extension of RoPE to enable token-level interaction among parallel reasoning sequences. It also proposes initialization and fine-tuning procedures for collaborative generation.

Critical strengths

The design provides a compact and flexible way to enable cross-sequence interaction within existing Transformer architectures.

Experiments include fine-tuned independent-generation baselines, multiple lane counts, and device-level timing measurements.

Critical weaknesses

The incremental benefit from collaboration is modest and its robustness remains unclear. In the 7B model, maj@4 improves by only 1.1 percentage points from N=1 to N=4 under the same LaneRoPE checkpoint, while Pass@1 barely changes. The 1.5B KTO model does not benefit from additional lanes. No training-seed variability or confidence intervals are reported.

Eq. (20) appears mathematically inconsistent with the intended KTO objective. Its negative branch has a negative slope, potentially reversing the intended optimization direction. If implemented as written, this would affect the interpretation of the KTO experiments.

Table 1 contains inconsistent averages. For example, the 7B HogWild! results average to approximately 42.6, rather than the reported 21.8.

Questions for authors

Is Eq. (20) a sign error? Please clarify the exact implemented function and the final optimization objective, including its optimization direction.

Are the collaboration gains robust across training seeds? Can the contribution of lane-aware positional encoding be isolated from cross-lane attention itself, for example by evaluating a matched model with lane frequencies set to zero?

2 comments

We thank the reviewer for the useful feedback.

  • Eq. 20 is mathematically inconsistent this was indeed a sign typo, which we corrected in the new version of the manuscript.

  • Inconsistent averages in Table 1 we mistakenly reported the average over 3 out of 4 datasets in two rows. This has been corrected now.

  • Can the contribution of lane-aware positional encoding be isolated from cross-lane attention itself, for example by evaluating a matched model with lane frequencies set to zero? We now include this LaneRoPE(Zero) baseline in the ablation study in Table 4. Our LaneRoPE(NTK*) generally outperforms this baseline on both SFT and NTK data (averaged across three math datasets, maj@4 55.355.3 vs 54.254.2 with KTO and 47.947.9 vs 44.844.8 with SFT); moreover, in the SFT setting, while sharing similar performance with N=1N=1 lanes (maj@4 45.945.9 vs 46.146.1), adopting N=4N=4 lanes hurts LaneRoPE(Zero) but favors our LaneRoPE(NTK*). These observations support a benefit of lane-aware positional encoding beyond cross-lane attention.

  • Robust across training seeds? We agree that evaluating multiple training seeds would strengthen the evidence. Due to the substantial computational cost of repeating training and corresponding evaluations, we could not quantify this variability. Still, for the 7B model, gains appear under both SFT and KTO, providing evidence across distinct training regimes.

I appreciate the authors' additional ablations, which help clarify the contribution of lane-aware positional encoding. I see potential in this direction, although the empirical gains from collaboration remain modest and their robustness is still unclear.

I believe this line of work could become more impactful if it leads to broader principles of collaborative parallel reasoning or insights that can be leveraged by other approaches. For now, I will maintain my original rating.

Reviewer_GPBBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces LaneRoPE, a positional-encoding scheme for multi-sequence, cross-lane collaborative generation with LLMs. The core idea is to extend RoPE from a 1D token index to a 2D (token index, lane/sequence index) grid, so that tokens generated in parallel across different sequences can attend to each other while preserving relative token and lane distances. The paper shows that GroupThink-style virtual indexing is a special case of LaneRoPE and that independent sampling can be approximated by an additional Fourier attention bias. It also proposes initialization strategies (GroupThink/NTK-aware correction, Fourier bias) and two fine-tuning recipes: SFT on synthetic collaborative traces and KTO on independently sampled correct/incorrect reasoning traces. Experiments on MATH500, AIME24/25, and AMC23 with DeepSeek-R1-Distill-Qwen 1.5B and 7B models show that trained LaneRoPE models improve majority-vote accuracy over sequential baselines and a reimplemented Bridge baseline, with modest overhead at inference.

Critical strengths
  1. The paper clearly distinguishes independent parallel sampling, adaptive branching, and fine-grained cross-lane collaboration. Showing that GroupThink and independent sampling are special/limiting cases helps unify the design space.

  2. The paper explains how cross-lane attention can be implemented by reshaping/packing lanes and reusing standard causal attention kernels. The FLOP and latency analysis argues that additional attention computation is negligible in memory-bound decoding regimes, which is a useful practical point.

  3. The synthetic collaborative SFT data recipe and KTO training on independently sampled reasoning traces are nontrivial and enable cross-lane behavior without RL. The ablation suggests KTO + NTK-aware initialization + trainable lane frequencies is the best configuration.

  4. For 7B models, LaneRoPE shows consistent improvements over the sequential baseline and usually over Bridge, especially under a fixed parallel budget. Fig. 4 suggests LaneRoPE composes with majority voting and scales better than independent sampling.

Critical weaknesses
  1. The main technical contribution is a more flexible parameterization of cross-sequence positions. GroupThink already performs cross-sequence attention via virtual positions, and Bridge already studies cross-lane attention. LaneRoPE’s advantage is flexibility and trainability, but the paper does not fully isolate how much gain comes from the positional encoding itself versus training data, KTO, and initialization.

  2. Experiments are restricted to mathematical reasoning benchmarks and two DeepSeek-R1-Distill-Qwen models. There is no evaluation on code, general reasoning, or instruction-following tasks, and no larger models. The 1.5B results are mixed—KTO breaks the multi-lane benefit—and the paper attributes this to small-model limitations without deeper analysis.

  3. The Bridge baseline is reimplemented rather than taken from official code, and the Hogwild! comparison uses a different base model, making direct claims about prior methods weaker. The paper is transparent about this, but it limits the strength of the “outperforms prior work” conclusion.

  4. SFT data is generated by prompting Qwen3-30B-A3B-Instruct to simulate collaborative assistants, then used to fine-tune DeepSeek-R1-Distill-Qwen models. The final SFT set is only 4,797 conversations after filtering, and KTO uses independently sampled traces rather than true collaborative traces. The dependence on synthetic data quality and the distribution mismatch are not fully analyzed.

  5. The packing implementation makes same-step tokens across lanes attend causally to each other, which introduces an asymmetry across lane order. The paper notes this is a convenient implementation choice, but does not measure whether lane order biases the learned collaboration or final answers.

  6. The paper acknowledges that it does not train a dedicated mechanism for merging outputs across lanes. Majority voting is used, but collaborative reasoning ideally should learn when to trust or merge lanes. This limits the claimed “collaborative reasoning” contribution

Questions for authors
  1. How much of the gain comes from LaneRoPE’s lane frequencies versus the synthetic SFT/KTO data and initialization? Can the authors train Bridge with the same data and parameter budget more thoroughly, or compare against a GroupThink model with the same trainable bias?

  2. Does the causal attention among same-step lanes create a systematic advantage for lower-index lanes? Can the authors randomize lane order at training or evaluation to test this?

  3. Do gains persist on code reasoning, general instruction following, or larger models? Is the method effective when N is larger than 4 or when lanes are heterogeneous?

1 comment

We thank the reviewer for the constructive feedback and we appreciate their recognition of our contributions on the training recipes and the practicality of our solution. Below, we try to address their main questions.

  • W1 Can we isolate gains from positional encoding versus data, KTO and initialization? Yes, Tab. 1 already isolates the benefit of LaneRoPE over the base model trained on the same data and the ablation in Tab. 4 isolates the impact of different initializations, as well as the advantage of KTO over SFT and no-training. To further isolate the benefits of lane positional encoding, we consider a new cross-lanes attention baseline without LaneRoPE ("LaneRoPE (Zero)") in Tab. 4. LaneRoPE (Zero) shows generally lower scores but also worsening performance with N=4N=4 lanes, especially on SFT (on SFT, maj@4 46.146.1 → 44.844.8 LaneRoPE (Zero) vs. 45.945.9 → 47.947.9 LaneRoPE(NTK*)). More details in Apx. D.2.

  • W2 Experiments are restricted to math benchmarks and small models. We focus on math as it offers verifiable reasoning and a simple voting strategy (unlike with coding or free-form responses, where voting techniques remain unclear). As our goal is to exploit parallelism in context of edge inference, larger models are outside the scope of this work (we now clarify this in the introduction).

  • W3a Bridge baseline reimplemented rather than taken from official code Bridge has no public code so our implementation is necessary for fair comparison.

  • W3b Hogwild! used different base models Our Hogwild results simply indicate its out-of-the-box performance in our evaluation setting, with same base models we adopted.

  • W4 Data quality dependency. We agree that sensitivity to synthetic-data quality deserve further study. Still, we believe the current experiments already provide evidence that our method is useful under two distinct training-data constructions. For SFT data, we chose a local 30B teacher as a practical compromise between the weaker native collaboration observed in smaller models (consistent with the observations in Hogwild!) and the generation cost of frontier LLMs. Moreover, our KTO experiments use the target model’s own independently sampled traces, with no distribution mismatch. While we do not fully isolate the effects of teacher choice and dataset size, our experiments and comparisons demonstrate useful adaptation within both data regimes. We leave exploring more scalable strategies to generate collaborative synthetic data as future work.

  • W5 Does lane-order bias the model? We find low correlations between lane-order and performance; enforcing full lane-order symmetry during training reduces performance. First, we note that training already shuffles completions to avoid fixed trace-to-position assignments. Also, the per-lane analysis in the table below finds weak dependency on the lane index (N=4 accuracies averaged over all four benchmarks). Moreover, Apx. D.3 now includes an experiment further randomizing the lane ids (used for LaneRoPE) to remove the lane-order asymmetry during training and finds worsening performance (on 7B model with NTK* and SFT data: 47.9→43.847.9 \to 43.8 maj@4).

ModelLane 1Lane 2Lane 3Lane 4
7B KTO52.6653.8152.8251.78
7B SFT42.7044.1443.4643.71
1.5B KTO41.4841.4041.3843.04
1.5B SFT35.6036.7233.5638.20
  • W6 Merging outputs across lanes Previous works like Bridge do not adopt such dedicated merging mechanisms either. This isolates cross-lane generation gains from dedicated aggregation design; moreover, we report pass@n which sets an upperbound on gains from them.

  • Q1a Gains from LaneRoPE vs training/initializations? Table 4 precisely ablates these initialization and training strategies.

  • Q1b Can you train Bridge with same data/parameters? All models use identical SFT or KTO data. We always tune all the weights in 1.5B models. Bridge adds layers so matching training parameters exactly requires non-obvious changes, but the difference is small (1.78B vs 1.86B). Due to GPU memory limits, we couldn't full-weights tune the 7B models. Apx. E documents our choice of only-Bridge-layers training after poor LoRA results; nevertheless, the parameter budget slightly favors Bridge (200M trainable parameters versus LaneRoPE's 90M with LoRA).

  • Q1c Can you train GroupThink? GroupThink is LaneRoPE(GT): the ablation in Tab. 4 already compares against GroupThink (with the same trainable bias) trained with both SFT and KTO.

  • Q2 Does lane-order bias the model? See answer to W5.

  • Q3a Evaluation scope See W2.

  • Q3b Larger N? Benefits for much larger values might be limited beyond the small-batch memory-bound regime. Some larger values are still worth exploring but require retraining with more synthetic data. Could the reviewer clarify what they mean by heterogeneous lanes?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.