acceptodds
Under review as a conference paper at ICLR 2027

Same Target, Different Paths: Learning from Multiple Reasoning Trajectories for Generative Recommendation

14 views

Abstract

Reasoning-based generative recommenders can support both explicit reasoning and direct Semantic ID (SID) generation within the same model. We study how multi-trajectory reasoning post-training affects this hybrid inference setting. In self-sampled rejection-sampling fine-tuning (RFT), multiple successful trajectories for the same request provide diverse reasoning paths while terminating at a shared target SID, coupling richer reasoning supervision with repeated terminal-target supervision. We introduce target-masked RFT (TM-RFT), which removes the terminal-target loss from augmentation trajectories while preserving supervision on their reasoning traces and on ordinary SFT data. On the OneReason benchmark, TM-RFT with up to four trajectories per request achieves a Weighted Pass@64 of 0.7576, compared with 0.6912 for SFT-only and 0.7011 for a group-normalized control that rescales the repeated target loss. Answer-supervised variants achieve higher standalone reasoning and direct-generation scores, but exhibit substantially stronger agreement between the two inference modes, with up to 44% of their candidates shared. In contrast, TM-RFT maintains route separation close to SFT-only while retaining 99.6% of the strongest single-mode performance under a fixed 64-candidate budget. On three public Amazon datasets, TM-RFT further improves two-route hit rate over both the original checkpoints and SFT-only.

Primary area:
generative models
Keywords:
Generative Recommendation, Large Language Models for Recommendation, Chain-of-Thought Reasoning, Rejection Sampling Fine-Tuning, Semantic IDs

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ALzfOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Target-Masked Rejection Sampling Fine-Tuning (TM-RFT) for generative recommendation. The method retains supervision on multiple successful reasoning trajectories while masking their shared terminal Semantic ID (SID) targets. This reduces redundancy between reasoning-based and direct recommendation, improving candidate diversity and overall retrieval performance. Experiments on OneReason and three Amazon datasets demonstrate the effectiveness of the proposed approach.

Critical strengths

Simple and effective methodology. The proposed masking strategy effectively addresses redundant target supervision in multi-trajectory learning.

Insightful analysis. The paper provides meaningful insights into candidate diversity and the complementarity of reasoning and direct decoding.

Empirical improvements. TM-RFT demonstrates performance gains across multiple recommendation benchmarks.

Critical weaknesses

Limited cold-start generalization. Reliance on predefined SIDs restricts scalability to unseen items and dynamic item catalogs.

Questionable practical competitiveness. Compared with recent approaches such as Rec-R1[1] and Rec-Zero[2], the practical advantages of this SID-based framework remain unclear.

Limited generalizability. Evaluation is restricted to Qwen3-based models and offline benchmarks, with insufficient analysis of computational costs and deployment feasibility.

[1] LIN, Jiacheng; WANG, Tian; QIAN, Kun. Rec-r1: Bridging generative large language models and user-centric recommendation systems via reinforcement learning. arXiv preprint arXiv:2503.24289, 2025.

[2] KONG, Xiaoyu, et al. Think before Recommendation: Autonomous Reasoning-enhanced Recommender. Advances in Neural Information Processing Systems, 2026, 38: 141209-141232.

Questions for authors

Scalability to Unseen Items: How can TM-RFT accommodate unseen or cold-start items that lack established Semantic IDs? Can the proposed framework generalize to dynamically evolving item catalogs without retraining?

Practical Competitiveness: Compared with recent reasoning-based recommendation approaches such as Rec-R1 and Rec-Zero, what are the practical advantages of TM-RFT? Could the authors provide comparisons in terms of recommendation performance, computational efficiency, and scalability?

Reviewer_x7oiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies post-training a Semantic-ID (SID)-generating recommender that supports two inference routes: explicit natural-language reasoning followed by an SID, or direct SID decoding. It observes that rejection-sampling fine-tuning (RFT) with multiple successful trajectories for the same user request repeatedly supervises a shared terminal target. The proposed Target-Masked RFT (TM-RFT) removes the domain-marker and three-SID-token loss from the self-sampled augmentation trajectories while keeping reasoning-token loss and full supervision on ordinary SFT examples. On the OneReason benchmark, the main k=4 setting improves Weighted Pass@64 from 0.6912 (SFT-only) to 0.7576 under a 32-reasoning/32-direct candidate allocation (Table 1); it also maintains low overlap between the two candidate routes. A secondary evaluation applies the method to three public Amazon categories using Evo-Rec initialization.

Critical strengths
  1. An interesting inference-aware question. Optimization that improves each separate reasoning/direct route need not improve their union under a fixed candidate count. Distinguishing single-route strength from complementary retrieval coverage is relevant for generative recommendation, especially because reasoning can be expensive and may not always improve accuracy.

  2. Simple and testable intervention. The loss decomposition explicitly distinguishes rationale-token supervision from the repeated terminal SID loss, with no architecture modifications. Keeping self-sampled trajectories and initialization fixed across Full-RFT, group-normalized Full-RFT, and TM-RFT is a good experimental design choice.

  3. The route-decomposition analysis is useful. The strongest standalone reasoning and direct results come from group-normalized Full-RFT (0.7081 and 0.7722), yet TM-RFT k=4 gives the strongest fixed-32+32 union. Table 3 shows a large overlap difference: around 12.4% for TM-RFT versus 43.6% for the group-normalized control. This directly motivates evaluating candidate complementarity, not only standalone hit rates.

Critical weaknesses
  1. The improvement cannot yet be attributed specifically to preventing repeated target supervision rather than changes in loss scale or example weighting. TM-RFT zeroes the entire terminal-target loss on augmentation rows, whereas Full-RFT retains it and the group-normalized control scales it by 1/n_g. These interventions also change the total gradient magnitude, relative amount of reasoning-token supervision, implicit weighting of easy requests that yield many accepted trajectories, and potentially the balance of CoT-format versus direct-format training. As the authors acknowledge, only 3,366 of 36,833 requests (9.14%) yield any successful reasoning trace, with 7,775 deduplicated accepted trajectories before capping (Section 5.2). Thus the augmentation is highly selective for requests on which the original model already succeeds. The most decisive missing control is supervising the answer for exactly one accepted trajectory per request while masking the rest, at otherwise fixed rationale loss and training-token budget. The discussion says such a control matters on Amazon, yet its numerical results are not supplied in the main tables. A sweep of fractional answer weights and a group-reweighted rationale loss would better distinguish the proposed mechanism from ordinary loss reweighting.

  2. The fixed '64-candidate budget' is not a fixed computation or latency budget. During augmentation sampling, reasoning trajectories are allowed up to 1,500 generated tokens (Section 5.2); the actual inference lengths are not reported, and reasoning candidates can cost far more than the four-token direct SID output. Therefore 32+32, 48+16, and 16+48 represent very different computational budgets, potentially by orders of magnitude. Even among models using the same split, answer-supervised variants can change average reasoning length. The claim that TM-RFT is better for hybrid inference is valid under the stated number-of-candidates budget, but not necessarily under equal latency, token generation, or serving cost. Please report average generated tokens, latency per request, distinct valid SID count, GPU inference cost, and a hit-rate versus compute/latency Pareto curve. Also compare against direct-only decoding with a compute-matched number of attempts or beam-search width. This is essential before suggesting a practical advantage for hybrid recommender inference.

  3. Key results are modest or mixed, and the paper lacks uncertainty estimates. The 0.7576 vs. 0.6912 difference on OneReason is encouraging, but it is a benchmark-weighted sum over four very different domains, not a percentage-point accuracy change. The best TM-RFT k differs across domains: k=4 on Video/Ads, k=1 on Product, and SFT-only remains best on Live (Table 1). The benchmark uses only one model family at 8B for its main experiment. The manuscript does not show uncertainty over model-training seeds, request-level confidence intervals/bootstraps, or variance from stochastic decoding. Results at k=8 regress substantially relative to k=4, also suggesting sensitivity. A small number of training runs may not be practical, but query-level paired bootstrap intervals and several candidate-sampling seeds are essential to tell whether the aggregate and domain-specific differences are reliable.

  4. The Amazon generalization claim is contradicted by one of the table entries and is not tested against the full set of variants. The abstract says that TM-RFT improves two-route hit rate over the released checkpoints and SFT-only "on three public Amazon datasets," which can be read as a per-dataset claim rather than a pooled-only claim. However, Amazon Games in Table 4 reports TM-RFT 22.30, below both Evo-Rec 22.42 and SFT-only 22.85. TM-RFT improves Office (26.37 vs. 24.79 SFT) and Industrial (25.53 vs. 24.06 SFT), giving a higher pooled hit rate (24.53 vs. 23.81), but the improvement is not universal. The statement should be corrected. More importantly, the discussion claims that on Amazon fully masked TM-RFT underperforms Full-RFT and group-normalized Full-RFT and that supervising one target per request closes the gap; however, those numerical comparisons do not appear in the supplied main PDF's Amazon table. This is potentially important negative evidence against the default α=0 choice, and should be reported prominently rather than only described qualitatively. The abstract should explicitly distinguish the pooled gain from the per-dataset results (two wins, one loss). It changes whether TM-RFT is a generally suitable training rule or a special case contingent on existing reasoning-conditioned answer supervision.

  5. The causal role and actual diversity of the 'reasoning trajectories' are insufficiently established. Rejection sampling filters by the final SID alone; an accepted rationale may be post-hoc, generic, or not causally responsible for reaching the correct item. Deduplicating only exact strings does not establish distinct reasoning paths. The paper measures candidate overlap between inference routes, not semantic diversity or usefulness of the sampled rationales themselves. A compelling mechanism test would compare embeddings/semantic clusters of successful CoTs, relate diversity to downstream gains, and contrast TM-RFT with a matched-token non-reasoning augmentation or one-trajectory control. The selection bias toward easy high-success-rate requests should be examined by request difficulty and domain. If performance gains arise mostly from increased input coverage or route diversification rather than reasoning, the conclusion should say so explicitly.

Questions for authors
  1. Can the authors provide the one-target-per-request + mask-the-rest ablation, fractional target-loss weights, and request-level rationale-weighting controls, including their Amazon results?

  2. What is the fixed-latency or fixed-decoded-token Pareto frontier for reasoning-only, direct-only, and hybrid inference? Does the 32+32 benefit survive when compute is the budget rather than candidates?

  3. What are paired bootstrap confidence intervals per domain and on Weighted Pass@64? How robust are the results to random sampling seeds and to the k=4 choice?

  4. Why does TM-RFT fail on Amazon Games, and which data regimes require retaining some reasoning-conditioned terminal-target loss? Are accepted rationale trajectories semantically diverse and faithful?

1 comment

We thank the reviewer for the constructive feedback. We have substantially revised the manuscript with additional ablations, statistical analysis, and cross-dataset evaluations.

1. Target supervision ablations (W1, Q1).
We added Half Weight (α=0.5 on each retained augmentation) and One Anchor (unit target weight on one retained augmentation per augmented request–target pair), keeping initialization, sampled trajectories, and reasoning-token supervision unchanged. TM-RFT (k=3) achieves 0.7893, outperforming Full-RFT (0.6460), Half Weight (0.7251), Group (0.7011), and One Anchor (0.7285). Full masking outperforms all tested schemes that retain terminal-target supervision on the added trajectories, although gradient-scale effects are not fully isolated.

2. Latency and practical applicability (W2, Q2).
We agree that generating CoT incurs additional latency compared with direct decoding. Our evaluation follows the official fixed-candidate protocol, not a fixed-compute budget. Importantly, TM-RFT improves all inference routes over SFT-only: reasoning-only (0.6396→0.7129), direct-only (0.7131→0.8099), and hybrid (0.6912→0.7893).

At a fixed budget of 64 candidates on OneReason, the best direct-only score (0.8099) exceeds the best mixed score (0.8035). However, industrial studies such as OneReason and OneLatent[1] demonstrate stronger or more efficient reasoning-based recommendation. TM-RFT is a training-time strategy that could complement these systems and potentially provide further improvements, although direct integration remains future work.

3. Statistical reliability (W3, Q3).
We added 50,000 paired bootstrap resamples (Appendix F). At 32+32, TM-RFT (k=3) improves Weighted Pass@64 over SFT-only by 0.0981 (pointwise paired 95% CI [0.0574, 0.1394]). The bootstrap resamples test requests with the evaluated checkpoints and predictions held fixed. We also report sensitivity across k∈{1,2,3,4,8}. Training-seed and decoding-seed variability remain limitations.

4. Amazon generalization (W4, Q4).
Updated experiments show TM-RFT outperforming SFT-only across all three Amazon datasets: Games (23.66 vs. 22.85), Office (26.99 vs. 24.79), and Industrial (26.33 vs. 24.06). Pooled coverage improves from 23.81% to 25.49%. Full-RFT, Group, and One Anchor comparisons are also included.

5. Reasoning diversity and success-conditioned selection (W5, Q4).
We added semantic clustering of successful trajectories (Appendix H). Across the full OneReason RFT dataset, 77% of groups with exactly four retained RFT trajectories contain at least one additional core reasoning path beyond the original SFT rationale. Appendix G further shows that successful augmentations concentrate on easier requests and frequent targets, supporting our motivation to avoid repeated target supervision. We acknowledge that semantic diversity alone does not establish causal faithfulness.

Overall, the revised evidence shows that TM-RFT improves recommendation performance across reasoning, direct, and hybrid inference by learning from additional reasoning trajectories without repeatedly supervising their shared terminal target.

References

[1] Hao Jiang et al. OneLatent: Latent Reasoning for Efficient Foundation Recommendation Models. arXiv:2610.30675, 2026.

Reviewer_euPGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how multiple successful reasoning trajectories with a shared target SID affect reasoning and direct decoding in generative recommendation. TM-RFT masks the terminal-target loss on augmented trajectories while retaining rationale supervision and ordinary SFT supervision. On OneReason, it improves the 32+32 candidate union over SFT and answer-supervised RFT controls, with substantially less agreement between the two routes. The strongest direct-only result remains group-normalized Full-RFT. Experiments continuing Evo-Rec checkpoints improve pooled two-route coverage on three Amazon datasets, although Games does not improve.

Critical strengths
  1. The controlled comparison is useful: Full-RFT, group-normalized Full-RFT, and TM-RFT share the augmentation pool, initialization, and optimization recipe. Tables 1 and 3 show that simply rescaling repeated target supervision does not recover the mixed-route coverage obtained by masking. This isolates a meaningful training choice beyond adding more generated data.

  2. The route analysis offers an informative diagnostic contribution. The group-normalized control has stronger standalone routes, while both answer-supervised controls have roughly 44% candidate overlap, versus 12.4% for TM-RFT at k=4 (Tables 2–3). Reporting several budget allocations and explicitly acknowledging the stronger single-route baseline makes the accuracy/diversity tradeoff easier to assess.

Critical weaknesses
  1. The demonstrated advantage is limited to a prescribed two-route protocol. In Table 2, group-normalized Full-RFT with 64 direct candidates scores 0.7722, above every tested TM-RFT mixed allocation (best: 0.7683). The paper acknowledges this, but does not establish when splitting the budget is preferable to using that stronger route. A matched-cost comparison or a concrete requirement for two-route retrieval would clarify the significance of the proposed training change; candidate count alone does not measure reasoning cost.

  2. The available public-data evidence leaves an important condition for masking unresolved. Section 7 states that fully masked TM-RFT underperforms Full-RFT and its group-normalized variant on Amazon, and that supervising one trajectory's target closes the gap, but Table 4 omits these comparisons. The conclusion also claims the highest reasoning-route HR@10 without corresponding results in the available PDF. These results are central to identifying when masking helps. The manuscript links no repository for the authors' implementation, and I could not access the supplementary files referenced on p. 9. Making those comparisons and the training/evaluation scripts available would substantially improve verifiability.

Questions for authors
  1. What use case requires splitting the candidate budget between both routes? Can TM-RFT outperform group-normalized Full-RFT's direct-only decoding at matched latency or generated-token cost? This would strengthen the practical significance beyond the fixed mixed protocol.

  2. Could you provide the Amazon results discussed in Section 7 for Full-RFT, its group-normalized version, and the variant supervising a target on one trajectory per request, including route overlap, reasoning HR@10, and query-level uncertainty? These comparisons would determine whether the masking conclusion generalizes beyond OneReason's SFT composition.

  3. Could you provide an accessible implementation or supplementary archive specifying the OneReason SFT data composition, trajectory-sampling checkpoint, and exact decoding and loss-masking procedures? This would let readers verify the reported controlled comparison.

1 comment

We thank the reviewer for the constructive feedback and recognition of our controlled comparisons and route-level analysis. We have substantially strengthened the manuscript to address the concerns.

1. Practical significance and inference latency (Weakness 1, Q1)

TM-RFT is a training-time method that improves multiple inference modes, including direct-only decoding, which requires no CoT generation.

Our updated Table 2 shows improvements from group-normalized Full-RFT to TM-RFT (k=3k=3) under matched candidate allocations:

  • Reasoning@64: 0.7081 → 0.7129
  • Direct@64: 0.7722 → 0.8099
  • Mixed 32+32: 0.7011 → 0.7893
  • Mixed 16+48: 0.7216 → 0.8035

The reviewer correctly noted that group-normalized Direct@64 (0.7722) exceeded the best TM-RFT mixed result in the original submission, which used k=4k=4. Our CoT analysis (Appendix H) covered all groups in the OneReason RFT dataset with exactly four retained RFT rationales. In each group, the four rationales added at most two distinct core reasoning paths beyond the original SFT rationale.This motivated us to evaluate a smaller cap of k=3k=3. With this cap, both the 32+32 and 16+48 mixed results exceed 0.7722.

Regarding latency, explicit CoT generation requires additional autoregressive decoding and is generally slower than direct inference. TM-RFT improves direct-only performance without introducing CoT decoding overhead. Our mixed-versus-direct comparisons use a fixed candidate count. A comparison at matched latency remains open.

On OneReason, the best direct-only result (0.8099) remains higher than the best mixed result (0.8035) at the same candidate count. Industrial systems such as OneReason and OneLatent use reasoning-enhanced retrieval alongside existing retrieval pipelines [1, Sec. 3.4.4]. Whether TM-RFT improves such systems in production remains to be evaluated.

2. Additional experiments and masking effectiveness (Weakness 2, Q2)

We have expanded the public-dataset experiments to include the requested group-normalized and One Anchor controls.

On Amazon (Table 3), TM-RFT achieves the highest pooled U(32+32) of 25.49%, compared with 23.81% for SFT-only, 23.16% for Full-RFT, 23.18% for group-normalized Full-RFT, and 23.64% for One Anchor. It leads on all three datasets, including Games, addressing the previous negative result.

We additionally report:

  • Candidate overlap (Table A5): TM-RFT reduces pooled overlap to 45.40%, compared with 65.86% for Full-RFT, 48.28% for group-normalized Full-RFT, and 48.58% for One Anchor.
  • Reasoning HR@10 (Tables A8 and A1): TM-RFT improves pooled Amazon HR@10 over Full-RFT from 13.25% to 13.83%, with gains persisting under reordered training.
  • Request-level uncertainty (Appendix F): The pooled improvement over SFT-only is +1.68 percentage points, with a pointwise paired 95% bootstrap CI of [1.24, 2.11] percentage points.

We also clarify the difference in original SFT composition between OneReason and Amazon. In the matched Amazon comparisons, learning from the same successful reasoning trajectories while masking their terminal-target losses yields higher joint coverage than either of the two controls that retain a total augmentation target coefficient of one.

3. Reproducibility (Weakness 2, Q3)

We have expanded Sections 3–4 and Appendix C with the sampling checkpoint, trajectory acceptance criteria, training configurations, token-level masking rules, and decoding procedures.

The anonymous reproduction repository is publicly accessible and provides the sampling and training scripts for the controlled comparisons.

https://huggingface.co/anony-tmrft/tm-rft-reproduction

It provides the implementation resources needed to reproduce our controlled comparisons.

We believe these revisions address the reviewer's main concerns and clarify that TM-RFT's contribution extends beyond a fixed mixed-inference protocol.

References

[1] Hao Jiang et al. OneLatent: Latent Reasoning for Efficient Foundation Recommendation Models. arXiv:2610.30675, 2026.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Comments — Summary of Major Revisions

We sincerely thank all reviewers for their valuable feedback. We have substantially revised the manuscript, adding new experiments, stronger controls, expanded analyses, and reproducibility materials. We summarize the major updates below.

1. RFT trajectory analysis and improved main results (Tables 1–2, Appendix H).
Our further analysis of RFT data revealed substantial semantic redundancy among successful reasoning trajectories. Across all eligible groups in the OneReason RFT dataset, four augmented trajectories contributed at most two additional core reasoning paths beyond the original SFT rationale, motivating us to investigate a smaller trajectory cap of k=3. The newly evaluated TM-RFT (k=3) achieves 0.7893 Weighted Pass@64, compared with 0.6912 for SFT-only and 0.7011 for group-normalized Full-RFT. It leads across all four recommendation domains and four of five tested candidate allocations, including reasoning-only (0.7129) and direct-only (0.8099). The improvements are therefore not limited to hybrid inference.

2. Stronger controlled comparisons (Tables 1–3).
We added Half Weight and One Anchor baselines, including the reviewer-suggested control that supervises one augmented target per request. All augmentation variants use matched training trajectories and optimization settings. We further expanded the decomposition of reasoning-only, direct-only, and shared hits to clarify the sources of retrieval gains.

3. Expanded cross-dataset validation (Table 3, Figure 4).
We added Full-RFT, group-normalized Full-RFT, and One Anchor comparisons on Amazon. TM-RFT now achieves the best results on all three Amazon datasets, improving pooled U(32+32) from 23.81% to 25.49% over SFT-only. Additional reasoning-route HR@10 experiments cover Amazon and Foursquare-NYC, including training-order replications.

4. Additional empirical analysis and uncertainty estimates (Appendices D–H).
We added paired bootstrap confidence intervals, domain-level route analyses, and measurements showing that successful RFT augmentations concentrate on easier requests and more frequent targets. We also expanded the analysis of retrieval strength, candidate overlap, and complementary coverage to better characterize how different supervision strategies affect the two inference routes.

5. Public release of model weights and reproducibility materials.
We have publicly released our trained model weights (checkpoints), together with training and evaluation code, reproduction scripts, and detailed experimental configurations through anonymous Hugging Face repositories. The reproduction repository is now accessible. We also clarified the distinction between candidate-count and computational budgets, along with the limitations of offline evaluation.

We appreciate the reviewers' constructive suggestions, which helped us strengthen both the empirical evidence and the presentation. We address the remaining reviewer-specific questions in our individual responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.