- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper considers the task of next-event prediction. Each event is characterized by a set of continuous and discrete features, which are predicted in parallel. The key component of the paper is that the authors propose to use two experts: (1) a pointer expert that makes predictions over categories encountered in the historical prefix; the numerical predictions are bounded within seen ranges; and (2) a direct expert that predicts over the whole categories’ vocabulary, and the numerical features can be extrapolated out of range. The competition between such experts allows achieving strong performance, even for features with a vocabulary of a high cardinality.
Critical strengths
The strength of the paper lies in the idea of balancing between prediction over the observed historical patterns and the generation of novel characteristics, unseen in the user's history, in the task of next-event prediction. It is an important problem if we are not to be biased towards just frequency-guided generation and when we are dealing with high-cardinality vocabularies.
Critical weaknesses
The critical weakness of the paper is that its main part is not self-consistent and contains limited experimental results that support the stated contributions. The main part contains too many references to the appendix, which hinders a comprehensive understanding of the proposed approach. Besides, the key stated contribution of retaining both mechanisms responsible for prediction over identities encountered in the history and the ones absent from the user prefix is not supported by the experiments. It is not clear whether the stated performance gains are obtained via better balancing of seen–unseen category prediction. The results section contains overly aggregated results over all baselines (Table 1), which makes the average benefits of the proposed approach over each existing method and their statistical significance obscure. Moreover, the contributions of the introduced architectural component, e.g., the Torus encoder or the two-expert setup, are not analyzed in the main part. Therefore, the absence of sufficient experimental results that justify the stated contributions and motivate the architectural choices, as well as the inconsistency of the main paper text because of heavy referencing to the appendix, make me advise rejecting the paper.
Questions for authors
-
How does the proposed method deal with labels outside the training vocabulary? In line 235, it is stated that there is a reserved unknown ID for such categories, but its usage at inference time is not clear.
-
Why do you not consider the baselines from the sequential recommendation domain (e.g., SASRec)? There are also a bunch of works which focus on next-item prediction, where items can have multiple attributes. Also, recommendation systems have the problem of big vocabularies and the need to recommend items unseen in the user's history.
-
Why is a simple classification head with an output dimension equal to the vocabulary size and operating on top of an embedding that encodes the user's history not enough?
-
How is the expert selected for making the prediction during inference? Why do we not use a mixed prediction that includes the output of each expert?
3 comments
Results summary
We have uploaded a revised paper that addresses your concerns and improves the clarity of the method and results. These tables are for your convenience; full details are in the revised paper.
Table A (Question 3). Large-vocabulary exact-label accuracy over five seeds, mean ± SE. Direct-only and pointer-only AP are independently tuned.
| Field | AP | Direct-only | Pointer-only | SASRec-style | Temporal MLP |
|---|---|---|---|---|---|
| IBM-CC city | 0.642 ± 0.00 | 0.613 ± 0.00 | 0.641 ± 0.00 | 0.233 ± 0.01 | 0.600 ± 0.00 |
| IBM-AML bank | 0.765 ± 0.01 | 0.426 ± 0.03 | 0.742 ± 0.00 | 0.0157 ± 0.00 | 0.503 ± 0.01 |
| SynSEPA IBAN | 0.103 ± 0.00 | 0.0408 ± 0.00 | 0.100 ± 0.00 | 0.00 ± 0.00 | 1.76e-3 ± 0.00 |
Table B (seen/unseen concern). AP mean exact-label accuracy over five seeds. Coverage is the fraction of targets present in the prefix.
| Field | Prefix coverage | Accuracy when prefix-present | Accuracy when training-unseen and prefix-present |
|---|---|---|---|
| IBM-CC city | 0.929 | 0.691 | 0.517 (70 targets) |
| IBM-AML bank | 0.961 | 0.796 | 0.849 (255 targets) |
| SynSEPA IBAN | 0.772 | 0.133 | 0.145 (5,678 targets) |
Prefix-absent accuracy is zero for both training-membership groups on all three fields.
Table C (component concern). Exact-label accuracy, mean ± SE over three seeds. This full-AP reference is separate from the five-seed AP above.
| Model | IBM-AML bank | SynSEPA IBAN |
|---|---|---|
| Full AP | 0.776 ± 0.00 | 0.102 ± 0.00 |
| Joint attention | 0.765 ± 0.01 | 0.106 ± 0.00 |
| Equal-width random codes | 0.760 ± 0.01 | 0.102 ± 0.00 |
| No score correction | 0.755 ± 0.00 | 0.0625 ± 0.00 |
Table D (individual-comparison concern). Mean ± SD over five seeds. Time: IBM-CC elapsed-time MAE in hours (lower is better). City: IBM-CC merchant-city accuracy; store: The Complete Journey store accuracy (higher is better).
Transformer copy and temporal MLP are supplementary controls. Displayed 0.00 reflects rounding; high denotes an SD string longer than ten characters.
| Method | Time | City | Store |
|---|---|---|---|
| AP | 23.2 ± 0.07 | 0.642 ± 0.00 | 0.676 ± 0.00 |
| SAHP | 43.9 ± 24.09 | 0.529 ± 0.01 | 0.664 ± 0.00 |
| RMTPP | 25.2 ± 0.04 | 0.572 ± 0.00 | 0.707 ± 0.00 |
| THP | 24.8 ± 0.15 | 0.601 ± 0.00 | 0.698 ± 0.00 |
| S2P2 | 25.5 ± 0.50 | 0.576 ± 0.01 | 0.691 ± 0.00 |
| NHP | 23.8 ± 0.05 | 0.617 ± 0.00 | 0.706 ± 0.00 |
| AttNHP | 24.0 ± 0.07 | 0.608 ± 0.00 | 0.682 ± 0.01 |
| IFTPP | 1.66e34 ± high | 0.608 ± 0.01 | 0.704 ± 0.01 |
| C-Diff | 54.0 ± 1.41 | 0.476 ± 0.00 | 0.539 ± 0.05 |
| Transformer copy | 23.6 ± 0.08 | 0.641 ± 0.00 | 0.730 ± 0.00 |
| SASRec-style | 24.0 ± 0.10 | 0.233 ± 0.01 | 0.688 ± 0.00 |
| Temporal MLP | 24.2 ± 0.07 | 0.600 ± 0.00 | 0.697 ± 0.00 |
| FlexTPP | 24.4 ± 0.13 | 0.538 ± 0.00 | 0.693 ± 0.00 |
Table E (Question 2). Complete SASRec-style comparison, mean ± SD over five seeds. Numerical fields use raw-unit MAE (lower is better); categorical fields use exact-label accuracy (higher is better). TCJ = The Complete Journey.
| Dataset | Field (metric/unit) | AP | SASRec-style |
|---|---|---|---|
| IBM-CC | Elapsed time (MAE/hours) | 23.2 ± 0.07 | 24.0 ± 0.10 |
| IBM-CC | Amount (MAE) | 37.3 ± 0.36 | 38.6 ± 1.20 |
| IBM-CC | MCC (accuracy) | 0.302 ± 0.00 | 0.274 ± 0.00 |
| IBM-CC | Merchant city (accuracy) | 0.642 ± 0.00 | 0.233 ± 0.01 |
| IBM-AML | Elapsed time (MAE/hours) | 2.87 ± 0.04 | 3.64 ± 0.39 |
| IBM-AML | Destination bank (accuracy) | 0.765 ± 0.02 | 0.0157 ± 0.00 |
| IBM-AML | Amount received (MAE) | 2.82e6 ± 230,972.24 | 4.11e6 ± 91,181.01 |
| IBM-AML | Receiving currency (accuracy) | 0.993 ± 0.00 | 0.986 ± 0.00 |
| IBM-AML | Payment format (accuracy) | 0.682 ± 0.00 | 0.533 ± 0.01 |
| MBD-Mini | Elapsed time (MAE/days) | 3.42 ± 0.04 | 3.58 ± 0.02 |
| MBD-Mini | Amount (MAE) | 6.66e5 ± 35.52 | 6.67e5 ± 79.51 |
| MBD-Mini | Event type (accuracy) | 0.567 ± 0.00 | 0.552 ± 0.00 |
| MBD-Mini | Event subtype (accuracy) | 0.566 ± 0.00 | 0.549 ± 0.00 |
| MBD-Mini | Source type (accuracy) | 0.922 ± 0.00 | 0.915 ± 0.00 |
| MBD-Mini | Destination type (accuracy) | 0.562 ± 0.00 | 0.546 ± 0.00 |
| TCJ | Elapsed time (MAE/days) | 5.25 ± 0.05 | 5.38 ± 0.02 |
| TCJ | Store (accuracy) | 0.676 ± 0.00 | 0.688 ± 0.00 |
| TCJ | Sales (MAE) | 22.3 ± 0.21 | 21.7 ± 0.04 |
| TCJ | Number of items (MAE) | 857 ± 4.20 | 879 ± 4.08 |
| SynSEPA | Elapsed time (MAE/days) | 1.66 ± 0.00 | 1.69 ± 0.02 |
| SynSEPA | Beneficiary IBAN (accuracy) | 0.103 ± 0.00 | 0.00 ± 0.00 |
| SynSEPA | Remittance category (accuracy) | 0.292 ± 0.00 | 0.292 ± 0.00 |
| SynSEPA | Amount (MAE) | 1.77e3 ± 2.99 | 1.77e3 ± 2.60 |
I appreciate the authors' additional experiments and the answers. While new results demonstrate the advantage compared to such an establish approach as SASRec, the positioning and narrative of the paper still remain unclear. If the authors focus on proposing a new method, than a more thorough discussion and comparison with other approaches on existing benchmarks is needed. To sum up, writing more consistent main part of the paper, identifying its clear contributions, and thorough comparison across existing benchmarks and methods would help paper become stronger. Now, the paper is far below an acceptance bar.
Thank you for the careful review. We have uploaded a revised paper that addresses your concerns and improves the clarity of the method and results. The result tables in our following reply are for your convenience; full details are in the revised paper.
1. Labels outside the training vocabulary (Question 1). At inference, labels outside the training vocabulary use the reserved unknown ID for that field. If the pointer selects that ID, it returns the first raw label in the observed history (the prefix) that carries it. The raw labels remain in the history. If unseen labels A and B appear in that order, selecting their shared ID returns A. Accuracy requires an exact raw-label match. Training membership is separate from prefix presence: a label unseen in training may already occur in the prefix. The separate novelty outcome means that the target's encoded ID is absent from the prefix. It does not name a new raw label or earn exact-label credit.
2. Sequential recommendation (Question 2). We added SASRec, adapted to predict the complete mixed event with causal attention, shared input/output category embeddings and numerical prediction heads. It can name training labels absent from the user's prefix. The comparison uses the same episodes, cutoffs, search budget and five final seeds. AP beats SASRec on 20 of 23 fields, including elapsed time on all five datasets and all three large-vocabulary fields. Table E gives every field.
3. Why not a vocabulary classifier (Question 3)? AP is more accurate than a vocabulary classifier on all three large-vocabulary fields (Table A). The comparison uses independently tuned direct-only AP with the same backbone. This classifier is already AP's categorical direct expert. It can name training-known labels outside the prefix. But a large vocabulary also creates a learning problem: a classifier learns an output weight vector for each label, and rare labels provide few positive examples for learning their vectors. The pointer learns a shared rule for finding useful historical evidence across labels. A rare label in the prefix can use this rule without needing its own learned output vector. This motivates complementing classification with copying.
A prefix also exposes far fewer candidates than the full vocabulary. AP combines evidence from repeated labels and learns a correction for their frequency advantage. Torus codes avoid learned per-category input vectors.
4. Expert selection and mixing (Question 4). Validation selects one expert per field before testing. That expert predicts every test example for the field. We select rather than mix to preserve each expert's output rule. Numerical and smaller categorical fields train both experts; large categorical fields use the pointer. Validation compares numerical error or label accuracy and smooths these measurements. Training weights scale each expert's gradient into the backbone; both private experts keep learning.
The categorical pointer scores prefix classes and a generic novel outcome; the classifier names training labels. Mixing requires a rule to match these outputs and turn the novel outcome into a named label. Numerical mixing also requires choosing whether to average transformed or raw values. We therefore compare complete experts using the same validation metric. Other expert selection rules and mixed predictions are useful future work. They require more nuanced experiments on output alignment, averaging and how the selection rule affects learning.
5. Seen/unseen prediction and the two-expert design (Critical weaknesses). AP can recover training-unseen labels from the prefix, and the expert study supports retaining both prediction mechanisms. Table B separates training membership from prefix presence and shows recovery under the decoding rule above. These three fields are pointer-only; this evidence tests historical reuse. The direct expert can name training-known labels absent from the prefix on eligible fields.
AP performs better overall than either independently selected single-expert control. The direct expert is stronger on some fields, while the pointer is stronger on others. This supports keeping both output rules available.
6. Components, individual results and presentation (Critical weaknesses). We revised the method and results for clarity, with clearer explanations of unknown labels and expert selection. We also added individual baseline comparisons and tests of the components and experts. Table C shows how the components affect prediction: axial attention and Torus improve bank accuracy over their controls, and score correction improves IBAN accuracy. Table D shows each baseline separately.