acceptodds
Under review as a conference paper at ICLR 2027

Factorized Coupled Sequence Modeling for Robust Sequential Prediction

Abstract

Sequential prediction in real-world systems often involves coupled sequence pairs, i.e. a primary action sequence and a co-evolving environment-level context sequence (e.g., app sub-interfaces and publisher sources, as opposed to item-intrinsic attributes like category or brand). Unlike item-level context, such environment-level context is susceptible to alignment noise: element-wise correspondence between the two sequences is inherently corrupted by delayed feedback and context switching. Existing approaches either ignore the context sequence or encode it jointly with actions at the element level. The former loses useful contextual signals, while the latter can introduce erroneous pairwise information and perturb the learned representations. We factorize the joint sequence model into two interacting sequence models that capture their own temporal dynamics and exchange information only at the representation level. We instantiate this design as Factorized Coupled Sequence Networks (FCSNet), with parallel sequence encoders, cross-sequence representation fusion, and dual prediction heads. Under mean-field posterior and factorized-prior assumptions, its training objective can be interpreted as an approximation to a factored evidence lower bound (ELBO) of the joint likelihood. We further introduce Conditional Contrastive Regularization (CCR), which uses similarity-adaptive weights to emphasize informative contrastive pairs under heavy-tailed sequential data. We validate FCSNet on two domains where coupled sequences with alignment noise naturally arise: e-commerce purchasing and online news browsing. Experiments on one public news benchmark and two industrial e-commerce datasets examine predictive performance and robustness to controlled alignment noise. Production deployment in an e-commerce search system further confirms real-world impact, achieving a 0.04 absolute-point CTR increase, 0.78% deal growth, and 0.64% GMV rise. Code is available in the anonymous repository at bluehttps://anonymous.4open.science/r/FCSNet-ForPublish/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
dual sequence prediction, channel feature, user behavior prediction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wkrZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces FCSNet for sequential prediction. It considers both user action sequences and evolving environmental context, which are not always well aligned. The method encodes the two sequences separately and combines their representations for prediction. It also includes dual prediction heads, adversarial prior regularization, and a similarity-adaptive contrastive objective (CCR). The experiments cover Outbrain and two large industrial datasets, with additional ablation studies and alignment perturbation tests. An online A/B test is also reported.

Critical strengths
  1. The problem seems well motivated. Separating user actions from environmental context is reasonable, particularly when the two sequences are not well aligned.
  2. The experimental evaluation is relatively comprehensive. The paper includes industrial datasets, ablation studies, and an online A/B test.
Critical weaknesses
  1. Figure 1 uses behavior-related icons to represent environmental context, which may be misleading. Besides, the overall architecture does not clearly show the dual prediction heads or the similarity-adaptive weighting mechanism of CCR.
  2. The captions of Tables 2 and 3 explicitly state that the checkpoint and test-split provenance of historical results “remains under verification.” Such an admission is inappropriate in a submitted paper, as the authors should verify these details before reporting the results.
  3. Most baselines are relatively outdated. (The latest method considered is from 2024 and appears only in the appendix.)
  4. It seems that the contributions of the auxiliary objectives are not sufficiently demonstrated. The ablation results show inconsistent benefits from LAPR and CCR. For example, removing LAPR improves NDCG@5 on Outbrain, while CCR provides only marginal gains on some datasets.
Questions for authors
  1. Could the authors clarify the use of behavior-related icons to represent environmental context in Figure 1? Would it be possible to better illustrate the dual prediction heads and the similarity-adaptive weighting mechanism of CCR?
  2. Why do the captions of Tables 2 and 3 state that the checkpoint and test-split provenance remains under verification?
  3. Could the authors include more recent baselines from 2025 and 2026?
  4. Could the authors further clarify the contributions of LAPR and CCR?
6 comments
AuthorsOpenReview ↗

Response summary of Reviewer wkrZ

Thank you for your careful review and valuable suggestions on the presentation and experimental evaluation. Your comments have guided the revised architecture figure, clearer reporting of the comparisons, the additional TCLARec comparison, and a more detailed explanation of the auxiliary objectives. We have also added three-seed KuaiSAR results. The response of each concern is listed below.

AuthorsOpenReview ↗

Response to Weakness1 and Question1 of Reviewer wkrZ

Thank you for pointing out the ambiguity. We have replaced ambiguous channel symbols with explicit page labels on both the input and output sides: Recom Page, Search Page, and Live Page. The revised figure shows the item head rV→oVr_V\to o_V and channel head rC→oCr_C\to o_C separately; both receive fused representations and contribute to LDSL\mathcal{L}_{\mathrm{DSL}}. The green and red CCR insets illustrate w+(h,h+)w^+(h,h^+) and w−(h,h−)w^-(h,h^-): lower-similarity positives and higher-similarity negatives receive larger weights, emphasizing harder pairs.

AuthorsOpenReview ↗

Response to Weakness2 and Question2 of Reviewer wkrZ

The statement about checkpoint and test-split provenance was introduced in error during AI-assisted proofreading. We take responsibility and have corrected both captions. All methods use the same fixed test partition within each dataset. Correcting the captions did not change the original numerical results. Thanks for pointing out this error, we have updated the manuscript.

Reviewer_JEo9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Factorized Coupled Sequence Networks (FCSNet) for sequential prediction with noisy action-context alignment. The method employs two independent Transformer encoders to model action and context sequences, followed by representation-level fusion and dual prediction heads. It further introduces adversarial prior regularization and Conditional Contrastive Regularization (CCR) with similarity-adaptive weights. The authors provide a factored-ELBO interpretation of the training objective and evaluate the approach on the public Outbrain dataset and two industrial e-commerce datasets. Additional experiments include ablation studies, controlled alignment-noise perturbations, and an online A/B test.

Critical strengths
  1. The paper addresses the important challenge of leveraging contextual information under noisy sequence alignment.

  2. The factorized dual-encoder design provides a simple and interpretable alternative to element-wise feature fusion.

  3. The paper reports experiments on public and large-scale industrial datasets, including robustness tests and online A/B evaluation.

Critical weaknesses
  1. Tables 2 and 3 explicitly acknowledge that some baseline checkpoints and test splits remain unverified, raising serious concerns about the reliability of the reported comparisons.

  2. The evaluation protocol differs between the paper and released code, including candidate ranking and Recall@K computation, while the testing implementation contains an undefined model attribute.

  3. The Outbrain context features are derived from document metadata rather than actual browsing environments, leaving the central assumption of naturally occurring action-context misalignment insufficiently supported.

  4. The proposed method primarily combines independent Transformer encoders, MLP-based late fusion, and existing regularization techniques, without sufficiently demonstrating advantages beyond simpler alternatives.

  5. The factored-ELBO derivation contains inconsistencies in probabilistic conditioning and expectations, while the implemented adversarial regularization does not correspond directly to the claimed KL-based objective.

Questions for authors

See Weaknesses

6 comments
AuthorsOpenReview ↗

Response summary of Reviewer JEo9

We sincerely thank you for the time and care you devoted to reviewing our work and for your constructive suggestions. Your comments have helped us improve the clarity of the experimental reporting, the suitability of the public dataset, and the explanation of the theoretical foundation. We address each of your concerns below.

AuthorsOpenReview ↗

Response to Weakness1 of Reviewer JEo9

The statement about checkpoint and test-split provenance was introduced in error during AI-assisted proofreading. We take responsibility and have corrected both captions. All methods use the same fixed test partition within each dataset. The caption correction itself did not change the original numerical results. Separately, we have replaced the Outbrain columns with results on the more suitable KuaiSAR dataset [1] for the 14 configurations reported in the manuscript, all using the same 2,570 test users and 999 fixed negatives per user. These KuaiSAR entries now report means over three training seeds (0, 1, 2), with mean ± sample standard deviation provided in our response to Reviewer jTq7, W4/Q3.

References

[1] Zhongxiang Sun et al. KuaiSAR: A Unified Search And Recommendation Dataset. CIKM, 2023.

AuthorsOpenReview ↗

Response to Weakness2 of Reviewer JEo9

Thank you for highlighting the protocol discrepancy. The original Outbrain sampled evaluator already ranked positives and negatives jointly, but the manuscript did not explicitly describe this or define capped Recall. We now state the candidate construction and Recall@K=H@K/min⁡(K,n+)\mathrm{Recall@K}=H@K/\min(K,n_+). For the added KuaiSAR [1] dataset, all methods share 2,570 test users and 999 fixed negatives per user. Next uses the first future item as the positive (1,000 candidates); Period uses distinct future items (999+n+999+n_+ candidates), where n+n_+ denotes the number of distinct positives. NDCG uses binary relevance, and both metrics are macro-averaged over users. For the industrial datasets, Period retains repeated positive entries within 10,000 total candidates. We have made these dataset-specific rules explicit. The legacy Outbrain implementation also averaged batch means equally, which differs from averaging all users when the final batch is smaller; the replacement KuaiSAR evaluation uses user-macro averaging.

Regarding the undefined attribute, we believe this refers to the auxiliary item-embedding export step:

if 'ours' in self.args.trainer_code:
    item_embedding = self.model.out_item.weight
else:
    item_embedding = self.model.out.weight

The self.model.out access reflects a compatibility issue with a development version. The reported baseline evaluations did not encounter this export incompatibility, so it did not affect their predictions or metrics. References

[1] Zhongxiang Sun et al. KuaiSAR: A Unified Search And Recommendation Dataset. CIKM, 2023.

Reviewer_jTq7OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses sequential prediction problems involving two coupled sequences: a primary action sequence and an environment-level context sequence, such as purchased items and shopping channels in e-commerce, or browsed articles and publisher sources in news recommendation. The authors argue that conventional approaches either ignore contextual sequences or combine actions and contexts through element-wise fusion, which can be vulnerable to alignment noise caused by delayed feedback and context switching. To overcome this limitation, they propose Factorized Coupled Sequence Networks (FCSNet), which independently encodes action and context sequences using parallel Transformer encoders and integrates their representations through cross-sequence fusion. The framework employs dual prediction heads for future actions and contexts, together with adversarial prior regularization motivated by an approximate factorized evidence lower bound (ELBO). Additionally, the authors introduce Conditional Contrastive Regularization (CCR), which assigns similarity-adaptive weights to positive and negative pairs to improve representation learning under sparse and heavy-tailed behavior distributions. Experiments on Outbrain and two large-scale industrial e-commerce datasets demonstrate improvements over several sequential recommendation baselines. Controlled perturbation experiments examine robustness to action–context alignment noise, while a seven-day production A/B test reports improvements in click-through rate, deals, and gross merchandise value.

Critical strengths
  • The paper has a well-motivated problem formulation with practical relevance, in which addresses the central challenge of learning from coupled behavioral and contextual sequences when their element-wise correspondence may be unreliable.
  • The proposed dual-encoder design is straightforward to implement and can potentially be applied to different sequential backbones. Separately modeling action and context dynamics before representation-level fusion provides a reasonable architectural inductive bias for reducing dependence on noisy element-wise correspondences.
  • Extensive experiments cover a public news dataset and two large-scale industrial datasets, including one with 10 million sequences. The reported improvements across next-item prediction and period-item prediction, together with a seven-day online A/B test, demonstrate practical applicability.
Critical weaknesses
  • Limited origninality: The proposed architecture primarily combines established techniques: parallel Transformer encoders, MLP-based representation fusion, auxiliary prediction heads, adversarial prior regularization, and contrastive learning with hardness-aware weighting. Although their combination is reasonable, the paper does not sufficiently distinguish FCSNet from existing multi-stream or late-fusion sequential modeling approaches. In particular, separating two sequences before fusion is a relatively standard architectural strategy. The paper should explain more precisely what is technically novel beyond applying late representation fusion to action–context sequences.
  • A central claim is that FCSNet is robust to alignment noise because the action encoder processes only the action sequence, making its pre-fusion representation invariant to context perturbations. However, the final action prediction still depends on the context representation through the fusion functions in Eq (3). Consequently, perturbations in the context sequence can still propagate to both fused representations and ultimately affect action predictions. The architecture guarantees only that the pre-fusion action representation is unchanged by context perturbations, not that the final predictions are robust.
  • CCR introduces similarity-adaptive weights that emphasize harder positive and negative pairs. However, hardness-aware contrastive learning and reweighting strategies have been widely studied, and the paper does not clearly establish the distinctive theoretical or algorithmic contribution of CCR. Furthermore, the expression in Eq(8) appears problematic: if each weighted term is interpreted literally as the displayed similarity multiplied by its softmax-normalized weight, there is no explicit contrastive log-ratio or positive-versus-negative normalization typical of InfoNCE-style objectives. The authors should clarify how this objective encourages positive agreement and negative separation, including its gradient behavior.
  • Tables 2 and 3 contain an explicit warning that some historical baseline rows have checkpoint and test-split provenance still under verification. This is a serious concern because the paper relies on these comparisons to establish its claimed superiority. Several baselines are also reported as out of memory on industrial datasets, leaving important comparisons incomplete. Furthermore, the manuscript does not provide statistical uncertainty estimates across repeated runs.
  • The Outbrain dataset has an average sequence length of only 2.36, which raises questions about how much temporal dependency can be learned by sequence encoders in that setting. The two industrial datasets are sampled from the same application and overlapping collection period, limiting their independence as evidence of cross-domain generalization.
Questions for authors

Q1. Can the authors clarify the mathematical relationship between the implemented objective and the proposed factored ELBO? Q2. What is the specific contribution of CCR compared with established hardness-aware contrastive objectives? Q3. Can the authors verify the provenance and statistical reliability of the offline comparisons (Table 2, 3)?

7 comments
AuthorsOpenReview ↗

Response to Weakness1 of Reviewer jTq7

Thank you for asking us to clarify the contribution. Our contribution centers on an application-driven insight: channel dynamics are part of user behavior and should be modeled together with action dynamics, while accounting for their potentially imperfect correspondence. We formulate this application problem, develop a design tailored to it, and provide a user-behavior likelihood interpretation.

Application insight and modeling rationale. Application channels have their own temporal dynamics and may be imperfectly aligned with item actions. This relatively underexplored setting motivates FCSNet to model channel history as a coupled behavioral sequence, using separate encoding, representation-level interaction, and dual prediction. Under the stated assumptions, Proposition 1 and Appendix A provide a user-behavior likelihood interpretation: a factored lower bound on the joint log-likelihood of future actions and channels motivates dual prediction and branch-wise prior regularization.

Empirical support. On AllChannelPay-10m (Table 2), next-item Recall@5 rises from 0.1443 for the joint-encoding control FCSNet-- to 0.1712 for FCSNet (+18.64% relative). Even without APR and CCR, the factorized model reaches 0.1680 (+16.42% over FCSNet--), supporting the factorized design even without either regularizer. The seven-day online A/B test yields +0.04 CTR percentage points, +0.78% deals, and +0.64% GMV, with positive relative CTR uplift across all displayed categories (Figure 2).

AuthorsOpenReview ↗

Response to Weakness2 of Reviewer jTq7

We agree with this distinction. For fixed model parameters, the pre-fusion action representation satisfies

hV=fV(V),hV(V,C)=hV(V,C~).h_V=f_V(V),\qquad h_V(V,C)=h_V(V,\widetilde C).

The final predictor has the form F(fV(V),fC(C))F(f_V(V),f_C(C)), so changes in CC can still affect its output through fusion. The architectural guarantee therefore applies only to the pre-fusion representation.

We have removed the corresponding robustness claims. The structural comparisons support the factorized modeling choice, and the online A/B test supports its practical utility. We have revised the manuscript to make this scope explicit.

AuthorsOpenReview ↗

Response to Weakness3/Question2 of Reviewer jTq7

Contribution. CCR performs soft hard-pair mining: less similar positive pairs and more similar negative pairs receive larger conditional weights. The positive and negative groups are normalized separately, with distinct temperatures, and CCR is applied to both action and context representations. Hard-pair mining is established; CCR provides a specific regularization design for coupled sequence modeling.

Absence of an explicit contrastive log-ratio or positive-versus-negative normalization. For a given anchor, let ai=si+a_i=s_i^+ and bj=sj−b_j=s_j^- denote the temperature-scaled similarities. Eq. (8) can be written as

ℓCCR=−∑iwi+ai+∑jwj−bj,w+=softmax⁡(−a),w−=softmax⁡(b).\ell_{\rm CCR}=-\sum_iw_i^+a_i+\sum_jw_j^-b_j,\qquad w^+=\operatorname{softmax}(-a),\quad w^-=\operatorname{softmax}(b).

With H(w)=−∑kwklog⁡wkH(w)=-\sum_k w_k\log w_k, the softmax definitions yield

ℓCCR=log⁡∑ie−ai+log⁡∑jebj−H(w+)−H(w−).\ell_{\rm CCR}=\log\sum_i e^{-a_i}+\log\sum_j e^{b_j}-H(w^+)-H(w^-).

This expresses CCR as a separately weighted attraction–repulsion objective. The entropy terms depend on the similarities, and the derivatives below account for this dependence.

Positive agreement, negative separation, and gradient behavior. Increasing all positive similarities by δ\delta, or decreasing all negative similarities by δ\delta, preserves the weights and decreases the loss by δ\delta.

More specifically, let μ+=∑iwi+ai\mu_+=\sum_iw_i^+a_i and μ−=∑jwj−bj\mu_-=\sum_jw_j^-b_j. Differentiating the displayed objective gives

∂ℓCCR∂ai=−wi+(1+μ+−ai),∂ℓCCR∂bj=wj−(1+bj−μ−).\frac{\partial\ell_{\rm CCR}}{\partial a_i}=-w_i^+(1+\mu_+-a_i),\qquad\frac{\partial\ell_{\rm CCR}}{\partial b_j}=w_j^-(1+b_j-\mu_-).

For hard positives with ai≤μ+a_i\le\mu_+, the derivative is negative; for hard negatives with bj≥μ−b_j\ge\mu_-, it is positive. Under gradient descent, these signs encourage higher similarity for hard positive pairs and lower similarity for hard negative pairs. For these hard pairs, the correction factors are at least one. Together with the larger conditional weights, they strengthen the optimization signals for positive agreement and negative separation.

The manuscript now includes the complete derivation. The existing weighting ablations assess the empirical contribution of these conditional weights.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.