acceptodds
Under review as a conference paper at ICLR 2027

Axial Pointer: Global-to-Local Prediction for Mixed-Type Event Sequences

Abstract

Real-world event sequences often record the next event as a complete set of interacting attributes, including irregular timing, numerical measurements, and categorical labels. Predicting that event is difficult because these output types behave differently, and because a categorical attribute may have thousands of possible values across a dataset even though only a small subset appears in one observed history. We introduce *Axial Pointer* (AP), which predicts all attributes from the same history and learns which earlier events provide useful evidence for each prediction. AP can reuse relevant categories or numerical values from that history while retaining the flexibility to predict when historical reuse is insufficient. In this way, AP uses a common source of historical evidence for numbers and categories without forcing these different outputs to be predicted in the same way. Across five heterogeneous transaction datasets and 23 predicted attributes, AP achieves the best or tied-best average performance on 21 attributes; it also performs best among the available learned comparison methods on all three evaluated categorical attributes with thousands of possible values.

Primary area:
learning on time series and dynamical systems
Keywords:
event sequence modeling, next-event prediction, pointer networks

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_LXqmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper considers the task of next-event prediction. Each event is characterized by a set of continuous and discrete features, which are predicted in parallel. The key component of the paper is that the authors propose to use two experts: (1) a pointer expert that makes predictions over categories encountered in the historical prefix; the numerical predictions are bounded within seen ranges; and (2) a direct expert that predicts over the whole categories’ vocabulary, and the numerical features can be extrapolated out of range. The competition between such experts allows achieving strong performance, even for features with a vocabulary of a high cardinality.

Critical strengths

The strength of the paper lies in the idea of balancing between prediction over the observed historical patterns and the generation of novel characteristics, unseen in the user's history, in the task of next-event prediction. It is an important problem if we are not to be biased towards just frequency-guided generation and when we are dealing with high-cardinality vocabularies.

Critical weaknesses

The critical weakness of the paper is that its main part is not self-consistent and contains limited experimental results that support the stated contributions. The main part contains too many references to the appendix, which hinders a comprehensive understanding of the proposed approach. Besides, the key stated contribution of retaining both mechanisms responsible for prediction over identities encountered in the history and the ones absent from the user prefix is not supported by the experiments. It is not clear whether the stated performance gains are obtained via better balancing of seen–unseen category prediction. The results section contains overly aggregated results over all baselines (Table 1), which makes the average benefits of the proposed approach over each existing method and their statistical significance obscure. Moreover, the contributions of the introduced architectural component, e.g., the Torus encoder or the two-expert setup, are not analyzed in the main part. Therefore, the absence of sufficient experimental results that justify the stated contributions and motivate the architectural choices, as well as the inconsistency of the main paper text because of heavy referencing to the appendix, make me advise rejecting the paper.

Questions for authors
  1. How does the proposed method deal with labels outside the training vocabulary? In line 235, it is stated that there is a reserved unknown ID for such categories, but its usage at inference time is not clear.

  2. Why do you not consider the baselines from the sequential recommendation domain (e.g., SASRec)? There are also a bunch of works which focus on next-item prediction, where items can have multiple attributes. Also, recommendation systems have the problem of big vocabularies and the need to recommend items unseen in the user's history.

  3. Why is a simple classification head with an output dimension equal to the vocabulary size and operating on top of an embedding that encodes the user's history not enough?

  4. How is the expert selected for making the prediction during inference? Why do we not use a mixed prediction that includes the output of each expert?

3 comments

Thank you for the careful review. We have uploaded a revised paper that addresses your concerns and improves the clarity of the method and results. The result tables in our following reply are for your convenience; full details are in the revised paper.

1. Labels outside the training vocabulary (Question 1). At inference, labels outside the training vocabulary use the reserved unknown ID for that field. If the pointer selects that ID, it returns the first raw label in the observed history (the prefix) that carries it. The raw labels remain in the history. If unseen labels A and B appear in that order, selecting their shared ID returns A. Accuracy requires an exact raw-label match. Training membership is separate from prefix presence: a label unseen in training may already occur in the prefix. The separate novelty outcome means that the target's encoded ID is absent from the prefix. It does not name a new raw label or earn exact-label credit.

2. Sequential recommendation (Question 2). We added SASRec, adapted to predict the complete mixed event with causal attention, shared input/output category embeddings and numerical prediction heads. It can name training labels absent from the user's prefix. The comparison uses the same episodes, cutoffs, search budget and five final seeds. AP beats SASRec on 20 of 23 fields, including elapsed time on all five datasets and all three large-vocabulary fields. Table E gives every field.

3. Why not a vocabulary classifier (Question 3)? AP is more accurate than a vocabulary classifier on all three large-vocabulary fields (Table A). The comparison uses independently tuned direct-only AP with the same backbone. This classifier is already AP's categorical direct expert. It can name training-known labels outside the prefix. But a large vocabulary also creates a learning problem: a classifier learns an output weight vector for each label, and rare labels provide few positive examples for learning their vectors. The pointer learns a shared rule for finding useful historical evidence across labels. A rare label in the prefix can use this rule without needing its own learned output vector. This motivates complementing classification with copying.

A prefix also exposes far fewer candidates than the full vocabulary. AP combines evidence from repeated labels and learns a correction for their frequency advantage. Torus codes avoid learned per-category input vectors.

4. Expert selection and mixing (Question 4). Validation selects one expert per field before testing. That expert predicts every test example for the field. We select rather than mix to preserve each expert's output rule. Numerical and smaller categorical fields train both experts; large categorical fields use the pointer. Validation compares numerical error or label accuracy and smooths these measurements. Training weights scale each expert's gradient into the backbone; both private experts keep learning.

The categorical pointer scores prefix classes and a generic novel outcome; the classifier names training labels. Mixing requires a rule to match these outputs and turn the novel outcome into a named label. Numerical mixing also requires choosing whether to average transformed or raw values. We therefore compare complete experts using the same validation metric. Other expert selection rules and mixed predictions are useful future work. They require more nuanced experiments on output alignment, averaging and how the selection rule affects learning.

5. Seen/unseen prediction and the two-expert design (Critical weaknesses). AP can recover training-unseen labels from the prefix, and the expert study supports retaining both prediction mechanisms. Table B separates training membership from prefix presence and shows recovery under the decoding rule above. These three fields are pointer-only; this evidence tests historical reuse. The direct expert can name training-known labels absent from the prefix on eligible fields.

AP performs better overall than either independently selected single-expert control. The direct expert is stronger on some fields, while the pointer is stronger on others. This supports keeping both output rules available.

6. Components, individual results and presentation (Critical weaknesses). We revised the method and results for clarity, with clearer explanations of unknown labels and expert selection. We also added individual baseline comparisons and tests of the components and experts. Table C shows how the components affect prediction: axial attention and Torus improve bank accuracy over their controls, and score correction improves IBAN accuracy. Table D shows each baseline separately.

AuthorsOpenReview ↗

Results summary

We have uploaded a revised paper that addresses your concerns and improves the clarity of the method and results. These tables are for your convenience; full details are in the revised paper.

Table A (Question 3). Large-vocabulary exact-label accuracy over five seeds, mean ± SE. Direct-only and pointer-only AP are independently tuned.

FieldAPDirect-onlyPointer-onlySASRec-styleTemporal MLP
IBM-CC city0.642 ± 0.000.613 ± 0.000.641 ± 0.000.233 ± 0.010.600 ± 0.00
IBM-AML bank0.765 ± 0.010.426 ± 0.030.742 ± 0.000.0157 ± 0.000.503 ± 0.01
SynSEPA IBAN0.103 ± 0.000.0408 ± 0.000.100 ± 0.000.00 ± 0.001.76e-3 ± 0.00

Table B (seen/unseen concern). AP mean exact-label accuracy over five seeds. Coverage is the fraction of targets present in the prefix.

FieldPrefix coverageAccuracy when prefix-presentAccuracy when training-unseen and prefix-present
IBM-CC city0.9290.6910.517 (70 targets)
IBM-AML bank0.9610.7960.849 (255 targets)
SynSEPA IBAN0.7720.1330.145 (5,678 targets)

Prefix-absent accuracy is zero for both training-membership groups on all three fields.

Table C (component concern). Exact-label accuracy, mean ± SE over three seeds. This full-AP reference is separate from the five-seed AP above.

ModelIBM-AML bankSynSEPA IBAN
Full AP0.776 ± 0.000.102 ± 0.00
Joint attention0.765 ± 0.010.106 ± 0.00
Equal-width random codes0.760 ± 0.010.102 ± 0.00
No score correction0.755 ± 0.000.0625 ± 0.00

Table D (individual-comparison concern). Mean ± SD over five seeds. Time: IBM-CC elapsed-time MAE in hours (lower is better). City: IBM-CC merchant-city accuracy; store: The Complete Journey store accuracy (higher is better).

Transformer copy and temporal MLP are supplementary controls. Displayed 0.00 reflects rounding; high denotes an SD string longer than ten characters.

MethodTimeCityStore
AP23.2 ± 0.070.642 ± 0.000.676 ± 0.00
SAHP43.9 ± 24.090.529 ± 0.010.664 ± 0.00
RMTPP25.2 ± 0.040.572 ± 0.000.707 ± 0.00
THP24.8 ± 0.150.601 ± 0.000.698 ± 0.00
S2P225.5 ± 0.500.576 ± 0.010.691 ± 0.00
NHP23.8 ± 0.050.617 ± 0.000.706 ± 0.00
AttNHP24.0 ± 0.070.608 ± 0.000.682 ± 0.01
IFTPP1.66e34 ± high0.608 ± 0.010.704 ± 0.01
C-Diff54.0 ± 1.410.476 ± 0.000.539 ± 0.05
Transformer copy23.6 ± 0.080.641 ± 0.000.730 ± 0.00
SASRec-style24.0 ± 0.100.233 ± 0.010.688 ± 0.00
Temporal MLP24.2 ± 0.070.600 ± 0.000.697 ± 0.00
FlexTPP24.4 ± 0.130.538 ± 0.000.693 ± 0.00

Table E (Question 2). Complete SASRec-style comparison, mean ± SD over five seeds. Numerical fields use raw-unit MAE (lower is better); categorical fields use exact-label accuracy (higher is better). TCJ = The Complete Journey.

DatasetField (metric/unit)APSASRec-style
IBM-CCElapsed time (MAE/hours)23.2 ± 0.0724.0 ± 0.10
IBM-CCAmount (MAE)37.3 ± 0.3638.6 ± 1.20
IBM-CCMCC (accuracy)0.302 ± 0.000.274 ± 0.00
IBM-CCMerchant city (accuracy)0.642 ± 0.000.233 ± 0.01
IBM-AMLElapsed time (MAE/hours)2.87 ± 0.043.64 ± 0.39
IBM-AMLDestination bank (accuracy)0.765 ± 0.020.0157 ± 0.00
IBM-AMLAmount received (MAE)2.82e6 ± 230,972.244.11e6 ± 91,181.01
IBM-AMLReceiving currency (accuracy)0.993 ± 0.000.986 ± 0.00
IBM-AMLPayment format (accuracy)0.682 ± 0.000.533 ± 0.01
MBD-MiniElapsed time (MAE/days)3.42 ± 0.043.58 ± 0.02
MBD-MiniAmount (MAE)6.66e5 ± 35.526.67e5 ± 79.51
MBD-MiniEvent type (accuracy)0.567 ± 0.000.552 ± 0.00
MBD-MiniEvent subtype (accuracy)0.566 ± 0.000.549 ± 0.00
MBD-MiniSource type (accuracy)0.922 ± 0.000.915 ± 0.00
MBD-MiniDestination type (accuracy)0.562 ± 0.000.546 ± 0.00
TCJElapsed time (MAE/days)5.25 ± 0.055.38 ± 0.02
TCJStore (accuracy)0.676 ± 0.000.688 ± 0.00
TCJSales (MAE)22.3 ± 0.2121.7 ± 0.04
TCJNumber of items (MAE)857 ± 4.20879 ± 4.08
SynSEPAElapsed time (MAE/days)1.66 ± 0.001.69 ± 0.02
SynSEPABeneficiary IBAN (accuracy)0.103 ± 0.000.00 ± 0.00
SynSEPARemittance category (accuracy)0.292 ± 0.000.292 ± 0.00
SynSEPAAmount (MAE)1.77e3 ± 2.991.77e3 ± 2.60

I appreciate the authors' additional experiments and the answers. While new results demonstrate the advantage compared to such an establish approach as SASRec, the positioning and narrative of the paper still remain unclear. If the authors focus on proposing a new method, than a more thorough discussion and comparison with other approaches on existing benchmarks is needed. To sum up, writing more consistent main part of the paper, identifying its clear contributions, and thorough comparison across existing benchmarks and methods would help paper become stronger. Now, the paper is far below an acceptance bar.

Reviewer_HR2JOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Axial Pointer combines axial attention with history-based and direct prediction heads to predict numerical and categorical event fields, reporting strong results across five transaction datasets.

Critical strengths
  • The model provides a unified approach to numerical and categorical prediction, with history-based copying offering a practical option for large categorical vocabularies.
  • Evaluation across five datasets, five random seeds, and ablations provides useful evidence of the approach's effectiveness.
Critical weaknesses
  • The figures are dense, and the main results table shows only AP's ranks and improvement ranges, making comparisons with individual baselines difficult without consulting the appendix.
  • The methodological contribution appears incremental, and the missing comparison with closely related models such as FlexTPP, leaves its advantage over existing mixed-type event models unclear.
Questions for authors
  • How much do axial attention, Torus encoding, and frequency correction each contribute beyond a standard Transformer with a copy head?
2 comments

Thank you for recognizing the value of unified mixed-type prediction and historical copying. We have revised the paper to address both critical weaknesses. The revision adds FlexTPP, a standard Transformer with copying, and separate tests of the three components you asked about. Tables A and B in the reply below give the supporting results.

1. Methodological contribution

AP's contribution is a common prediction mechanism for a complete mixed-type event. Each field learns which past events provide useful evidence. A categorical field combines evidence for repeated identities; a numerical field uses a weighted average of past values. The same historical-position interface therefore supports different output types, while attention across fields and time lets them share context.

This design connects identity representation, prediction, and output choice. Fixed-width Torus codes represent categories without learning a separate input vector for each one. The categorical pointer chooses among identities found in the history, and learned frequency correction controls how repeated occurrences affect that choice. On eligible fields, validation selects between historical reuse and direct prediction.

These mechanisms work together to address a practical mismatch: the full category vocabulary can be large, while one person's history contains a small set of relevant candidates. AP combines this local prediction strategy with a shared treatment of numerical and categorical fields. Table B below supports the value of that design through comparisons with both ordinary copying and a closely related mixed-type event model.

2. Contributions of axial attention, Torus, and frequency correction

Retaining axial attention, Torus encoding and frequency correction each improves destination-bank accuracy. Axial attention also reduces timing error across all five datasets, and frequency correction improves beneficiary-IBAN accuracy (Table A).

Table A below changes one component at a time within AP. Joint replaces axial attention with attention over the event–field grid. Rand replaces Torus with fixed random codes of the same width. No FC removes the repeated-class score correction while keeping class aggregation and training-loss frequency weights. The other AP mechanisms and selected settings are retained.

The controls also reveal which effects depend on the field: joint attention improves IBM-CC amount and SynSEPA IBAN. Table A therefore answers where the components help, alongside the complete-model comparison in Table B. These are separate component tests, not an additive breakdown of the difference from Transformer copying. The component study uses its own three-seed AP reference and common validation subset.

3. Advantage over FlexTPP and standard Transformer copying

FlexTPP is now part of the main comparison. It predicts a mixed event by generating its fields in sequence. AP predicts all fields from the same observed history. At evaluation, neither receives true fields from the event being predicted. The comparison uses the same data, scored targets, and training-update budgets, with separate validation searches and five final seeds.

The Transformer-copy control directly addresses your question about a standard Transformer with a copy head. It combines a conventional causal event encoder with categorical copying and direct numerical prediction. Its copy head already combines repeated labels, so the comparison tests AP against a model that can exploit recurrence.

Table B below shows AP's advantages over both baselines on merchant city, destination bank, and beneficiary IBAN. The numerical results matter too: AP improves received-amount prediction on IBM-AML and item-count prediction on The Complete Journey. These gains extend beyond categorical copying. Together, Tables A and B give complementary evidence: the component tests identify useful design choices, and the baseline comparisons establish their value as an integrated system.

4. Clearer presentation and direct comparisons

We have clarified the motivation and explained the figures alongside the revised method: the first introduces global identity representation and local prediction, the second shows how AP predicts a complete event, and the third compares categorical performance. The revised results text now discusses individual baseline comparisons and includes selected component results in the main text.

Tables A and B below make every field directly comparable without an appendix lookup. They show AP beside the alternatives in the original metrics, with uncertainty and the best means identified. This makes both the contribution and its empirical support easier to assess.

AuthorsOpenReview ↗

Results summary

Results supporting our reply above

All 23 fields are shown in each table. CC=IBM-CC; AML=IBM-AML; MBD=MBD-Mini; CJ=The Complete Journey; SEPA=SynSEPA. "acc" is exact raw-label accuracy (higher is better); all other rows use raw-unit MAE (lower is better). h=hours; d=days. Bold marks the best unrounded mean within each row; equal rounded values need not be ties. SD is sample standard deviation across seeds; SE=SD/√n. Uncertainties rounded to 0.00 are not necessarily zero.

Table A. Component interventions: mean ± SE, seeds 1–3

AP is this study's separate full-AP reference. Joint=joint causal attention; Rand=fixed random identity codes; No FC=no repeated-class score correction. All arms retain the other AP mechanisms and inherited settings, with the same fixed validation subset.

FieldAPJointRandNo FC
CC time(h)23.2±0.0723.3±0.0823.2±0.0923.2±0.04
CC amount37.1±0.2736.7±0.1337.3±0.3637.4±0.22
CC MCC acc0.302±0.000.300±0.000.301±0.000.302±0.00
CC merch.city acc0.642±0.000.642±0.000.641±0.000.640±0.00
AML time(h)2.81±0.022.85±0.052.87±0.032.76±0.02
AML bank acc0.776±0.000.765±0.010.760±0.010.755±0.00
AML received amt2.79e6±110,989.853.01e6±102,198.342.81e6±181,090.432.71e6±103,301.43
AML currency acc0.993±0.000.992±0.000.993±0.000.993±0.00
AML format acc0.683±0.000.681±0.000.679±0.000.679±0.00
MBD time(d)3.43±0.013.48±0.013.44±0.003.43±0.03
MBD amount6.66e5±67.886.66e5±85.296.66e5±289.866.66e5±289.32
MBD type acc0.568±0.000.565±0.000.568±0.000.567±0.00
MBD subtype acc0.565±0.000.563±0.000.565±0.000.565±0.00
MBD src.type acc0.922±0.000.922±0.000.922±0.000.922±0.00
MBD dst.type acc0.560±0.000.559±0.000.562±0.000.562±0.00
CJ time(d)5.23±0.025.38±0.025.27±0.015.28±0.01
CJ store acc0.666±0.000.672±0.010.666±0.010.679±0.00
CJ sales22.3±0.0322.6±0.1022.3±0.1722.1±0.08
CJ items856±0.87865±2.43858±1.38857±1.21
SEPA time(d)1.66±0.001.67±0.001.67±0.001.66±0.00
SEPA IBAN acc0.102±0.000.106±0.000.102±0.000.0625±0.00
SEPA remit acc0.293±0.000.292±0.000.292±0.000.292±0.00
SEPA amount1.77e3±2.101.77e3±1.311.77e3±4.081.77e3±3.61

Retaining axial attention, Torus codes, and score correction improves destination-bank accuracy by 1.14, 1.63, and 2.08 percentage points, respectively, using unrounded paired differences. Axial attention lowers time MAE on all five datasets. Score correction also improves IBAN prediction. Effects vary by field: joint attention leads on CC amount and SEPA IBAN.

Table B. Complete-model comparisons: mean ± SD, seeds 1–5

Copy=standard Transformer with copying; Flex=FlexTPP. AP is the same five-seed reference used in the revised main comparison. This reference is separate from Table A.

FieldAPCopyFlex
CC time(h)23.2±0.0723.6±0.0824.4±0.13
CC amount37.3±0.3636.9±0.3437.5±0.06
CC MCC acc0.302±0.000.269±0.010.237±0.00
CC merch.city acc0.642±0.000.641±0.000.538±0.00
AML time(h)2.87±0.042.97±0.053.42±0.01
AML bank acc0.765±0.020.745±0.010.0157±0.00
AML received amt2.82e6±230,972.243.88e6±264,592.374.32e6±1,349.74
AML currency acc0.993±0.000.993±0.000.974±0.00
AML format acc0.682±0.000.679±0.000.425±0.00
MBD time(d)3.42±0.043.48±0.013.44±0.01
MBD amount6.66e5±35.526.66e5±140.126.66e5±63.68
MBD type acc0.567±0.000.569±0.000.548±0.00
MBD subtype acc0.566±0.000.567±0.000.545±0.00
MBD src.type acc0.922±0.000.921±0.000.908±0.00
MBD dst.type acc0.562±0.000.563±0.000.530±0.00
CJ time(d)5.25±0.055.21±0.025.19±0.02
CJ store acc0.676±0.000.730±0.000.693±0.00
CJ sales22.3±0.2121.5±0.0321.5±0.03
CJ items857±4.20878±0.09874±3.03
SEPA time(d)1.66±0.001.68±0.001.66±0.02
SEPA IBAN acc0.103±0.000.0360±0.000.00±0.00
SEPA remit acc0.292±0.000.279±0.010.275±0.02
SEPA amount1.77e3±2.991.78e3±6.951.77e3±17.00

AP leads both Copy and Flex on merchant city, destination bank, and beneficiary IBAN; its numerical gains include AML received amount and CJ item count. These field-level comparisons support the integrated mixed-type design.

Reviewer_T5mCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Axial Pointer for one-step prediction of mixed numerical and categorical event records. An axial transformer shares context across fields and time. Pointer heads reuse historical values, aggregating repeated categorical identities and interpolating numerical values, while eligible axes also have direct prediction heads selected through a validation-based routing procedure. A deterministic Torus encoding provides fixed-width category representations. Experiments on five transaction datasets and 23 prediction axes report best or tied-best seed-averaged performance on 21 axes against the learned comparison methods. The appendix includes expert ablations, a causal mean/mode reference, and efficiency measurements.

Critical strengths
  1. The model addresses a concrete mismatch between large global vocabularies and small history-local candidate sets. Class-level aggregation of repeated positions and type-specific interpretation of pointer weights form a coherent approach to heterogeneous records, rather than forcing all fields into one output geometry.
  2. The empirical study is broad and transparent. It covers five datasets, 23 axes, five seeds, a causal mean/mode reference, expert ablations, and detailed resource accounting. The strong high-cardinality results and the reported failures on The Complete Journey make the evidence more informative than a single aggregate score.
Critical weaknesses
  1. The comparison is dominated by adapted temporal point-process models. It lacks a matched learned copy/explore or sequential-tabular baseline, as well as a strong time-aware tabular predictor adapted to the same history-to-next-event task. Such controls would help distinguish the benefit of AP’s design from historical reuse and temporal feature construction. Moreover, the direct-only and pointer-only ablations inherit AP’s selected configuration rather than receiving independent tuning, leaving the architectural advantage insufficiently isolated.
  2. The high-cardinality evidence does not characterize the method’s dependence on recurrence. Those axes are pointer-only: an absent target can be flagged as novel, but its identity cannot be predicted. Results use fixed 50-event episodes and do not break down candidate coverage or accuracy by prefix presence. This limits the interpretation of the global-to-local advantage when identities turn over or histories contain fewer repeated targets.
Questions for authors
  1. Are there results for a learned copy/explore baseline or a sequential-tabular transformer under the same input schema? For the expert ablations, do existing tuning results indicate whether their ranking changes when each variant selects its own configuration?
  2. Could you include a strong time-aware tabular baseline, such as MLP or TabM with temporal embeddings following [1], adapted to the same observed history and next-event targets? Please use matched information and tuning budgets, without revealing the next event’s true timestamp or fields. Also clarify how your task differs from temporal-shift prediction on [2].
  3. For each high-cardinality axis, what fraction of test targets appears in the observed prefix, and what is AP’s accuracy conditional on prefix presence? Please distinguish prefix-absent but training-seen targets from targets unseen during training.
2 comments

Thank you for the careful review. We have uploaded a revised paper addressing your concerns. Tables A–D in the following reply are only for your convenience; full results, uncertainty estimates and method details are in the revised paper.

Question 1 and Weakness 1: matched sequence models and independent tuning

We added a Transformer-copy control that combines repeated raw-label occurrences and tests learned historical reuse. We also added a SASRec-style causal Transformer adapted to every numerical and categorical field of the same input schema. Its full-vocabulary outputs can predict training-known labels absent from the prefix. Copy and MLP are author-constructed supplementary controls.

Table A shows AP leading these baselines on all three high-cardinality fields. It also beats copying on elapsed time in four datasets and on item count in The Complete Journey. Copying remains competitive elsewhere, but AP leads key fields even when learned reuse is available.

We independently selected width, depth and learning rate for each fixed-expert variant using AP's search space and budget. AP has the best mean among the three variants on 16 of 23 fields, versus 14 with inherited settings. Only two expert-versus-AP orderings reverse: pointer-only falls below AP on merchant city and beneficiary IBAN. Table A gives these results; both fixed experts retain fields on which they lead.

Table D tests axial attention, identity codes and frequency correction while retaining historical reuse and temporal inputs. Each retained component improves destination-bank accuracy; frequency correction also improves IBAN accuracy. These paired three-seed comparisons use inherited AP settings and one validation subset, with a separate AP reference. Effects vary by field and are not additive.

Together, the comparisons support AP's shared historical-position interface. Axial context links fields and time; fixed-width codes represent identities; pointers select categories or average numerical history. Validation selects pointer or direct prediction on eligible fields. This combined design is a promising contribution for mixed-type events.

Question 2 and Weakness 1: temporal features and the TabReD distinction

We adapted the requested MLP to this task with learned Fourier time embeddings, last values, numerical prefix means and categorical prefix frequencies. Its clock is time since the first observed event. All features use the prefix; no true successor time or field is supplied.

Each added baseline and fixed-expert variant has its own 16-attempt search, with the same trial and final update budgets as AP. Splits, targets, training-fitted transforms and five final seeds are shared. Selection uses validation only. Table A shows AP beating the MLP on elapsed-time MAE in every dataset and on all three high-cardinality fields.

TabReD tests whether models trained on earlier rows can predict a label or scalar on later rows as the data distribution changes. The test row's features are supplied. For example, cooking-time prediction receives the placed order's contents. Historical aggregates can also be features.

Our input is only an entity's observed event prefix. We predict the whole next event, including its waiting time, numerical values and categorical identities. Its features are unknown: both the next transaction amount and the time until it occurs must be predicted.

Time thus serves different roles: TabReD uses calendar order to assess distribution shift; we use observed inter-event gaps as context and predict the next gap. Our splits hold out entity-history windows, without a controlled calendar-shift comparison.

Question 3 and Weakness 2: recurrence and candidate coverage

Table B gives coverage and accuracy conditional on prefix presence for each high-cardinality field. Overall accuracy equals their product. Coverage measures whether the target is available to copy; conditional accuracy measures selecting it. IBAN coverage is much higher than conditional accuracy, so candidate availability alone does not make prediction easy. AP also improves over learned copying in Table A.

Table C separates all four prefix/training-membership groups. A training-seen but prefix-absent target is available to a vocabulary generator. A target absent from both sets is available to neither copying nor that generator. AP's pointer-only fields have zero exact-label accuracy in both absent groups. Novelty denotes encoded-class absence and does not name an external label.

Present but training-unseen hits are exact raw-label recoveries under the first-match rule: unseen labels share an unknown input ID, and AP returns its first matching raw prefix label. Tested prefixes contain 40–49 events within 50-event episodes. Shorter histories and controlled identity turnover remain to be evaluated. The revised claim concerns measured coverage and selection within that support.

AuthorsOpenReview ↗

Results summary

We have uploaded a revised paper addressing your concerns. These tables are only for your convenience. Full results, uncertainty estimates and method details are in that paper. Table letters here are local to our replies.

Table A. Questions 1–2 and Weakness 1: matched model and expert comparisons. Five-seed means. Time, amounts, sales and items: raw-unit MAE (lower is better); other fields: exact-label accuracy (higher is better). Bold marks the best unrounded mean among these seven methods; equal rounded values need not be ties. Copy: Transformer copying; SAS: SASRec-style; MLP: time-aware MLP; Flex: FlexTPP; D/P: independently tuned direct-only/pointer-only AP. CC: IBM-CC; AML: IBM-AML; MBD: MBD-Mini; TCJ: The Complete Journey; Syn: SynSEPA. Copy and MLP are author-constructed supplementary controls. The revised paper keeps these controls separate from its main comparison and reports uncertainty for all methods.

DatasetFieldAPCopySASMLPFlexDP
CCTime (h)23.223.624.024.224.423.323.4
CCAmount37.336.938.639.337.536.937.7
CCMCC0.3020.2690.2740.2880.2370.3030.299
CCMerchant city0.6420.6410.2330.6000.5380.6130.641
AMLTime (h)2.872.973.643.453.423.312.79
AMLDestination bank0.7650.7450.01570.5030.01570.4260.742
AMLReceived amount2.82e63.88e64.11e64.25e64.32e64.61e62.84e6
AMLReceiving currency0.9930.9930.9860.9870.9740.9890.992
AMLPayment format0.6820.6790.5330.5130.4250.6170.673
MBDTime (d)3.423.483.583.603.443.433.49
MBDAmount6.66e56.66e56.67e56.66e56.66e56.66e56.66e5
MBDEvent type0.5670.5690.5520.5560.5480.5670.557
MBDEvent subtype0.5660.5670.5490.5540.5450.5650.556
MBDSource type110.9220.9210.9150.9160.9080.9220.917
MBDDestination type110.5620.5630.5460.5490.5300.5610.550
TCJTime (d)5.255.215.385.395.195.185.15
TCJStore ID0.6760.7300.6880.6970.6930.6920.702
TCJTotal sales22.321.521.721.521.521.421.4
TCJNumber of items8578788791.41e13874880871
SynTime (d)1.661.681.691.831.661.671.66
SynBeneficiary IBAN0.1030.03600.0001.76e-30.0000.04080.100
SynRemittance category0.2920.2790.2920.2820.2750.2930.276
SynAmount1.77e31.78e31.77e31.78e31.77e31.78e31.77e3

Table B. Question 3 and Weakness 2: raw-label prefix coverage and AP accuracy (%). Accuracy is the five-seed mean; target counts are fixed across seeds.

FieldPresent / all targetsCoverageAccuracy given presenceOverall accuracy
CC city51008 / 5488092.969.164.2
AML bank92971 / 9675096.179.676.5
Syn IBAN38622 / 5000077.213.310.3

Table C. Question 3 and Weakness 2: prefix and training membership. Each cell: target count / five-seed mean conditional AP accuracy (%). Counts and accuracies use exact raw labels.

FieldPresent, training-seenPresent, training-unseenAbsent, training-seenAbsent, training-unseen
CC city50938 / 69.170 / 51.73844 / 0.028 / 0.0
AML bank92716 / 79.6255 / 84.93769 / 0.010 / 0.0
Syn IBAN32944 / 13.15678 / 14.5907 / 0.010471 / 0.0

Table D. Weakness 1: accuracy gain from retaining each component (percentage points). Differences use unrounded means from paired three-seed comparisons with inherited AP settings and a shared validation subset. Each change is compared with this study's own full-AP reference, separate from Table A's five-seed AP.

FieldComponent retainedAccuracy gain
AML bankAxial attention+1.14
AML bankTorus encoding+1.63
AML bankFrequency correction+2.08
Syn IBANFrequency correction+3.92

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.