- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies feature-based choice models, and proposes a new method called Consideration circuits. This is defined in terms of a multinomial logits (MNL) structured and applied at nodes of a directed cyclic graph. The paper shows that the representational capacity of this method exceeds a single MNL, and it can represent optimal choice for decoy effect and compromise task examples. They also provide more general representational capacity results for deterministic case, under simplifying assumptions/restrictions on convex combinations and translation. Empirically, they show gains against MNL and when use the proposed method on top of (or as the last layer of) other baselines.
Critical strengths
The proposed method is simple and the paper provides theory characterizing its representational power.
Critical weaknesses
The reviewer is skeptical about the long-term impact of the contributions. As a standalone model, the circuit is limited by the convex hull and translation constraints, whereas transformer-based choice models transform item features non-linearly and are not subject to these limits; the theoretical results compare circuits only with MNL, random-utility models, and circuits of other depths, not with these architectures. The paper’s remaining argument is that the circuit’s constrained structure gives better robustness, but this rests mainly on one dataset (Expedia), in a reduced setting.
Questions for authors
Can the authors justify why on Expedia, the paper’s ranking of models is the reverse of prior work on the same dataset (e.g., the TCNet and RUMnet papers), where deep models clearly outperform MNL, whereas here they all perform worse than MNL?
5 comments
(2/3) Response to Reviewer 5KAN
2. Expedia Evaluation and Model Rankings
Question: (1) The evidence for robustness rests mainly on one dataset (Expedia), in a reduced setting. (2) Why do deep models perform worse than MNL in the reported Expedia evaluation, whereas prior work, including the TCNet and RUMnet papers, finds that they outperform MNL?
Response: Thank you for pointing out the limitations and lack of clarity in our original evaluation. Addressing these questions has helped us make the comparison more comprehensive and clarify how empirical performance varies across evaluation settings. These concerns primarily relate to Table 3 of the original manuscript, which reported only one temporal evaluation on Expedia. We address them together below.
(1). Expanded Expedia evaluation.
In the revised Table 3 (Section 5.1, p. 9, lines 469–481), we have added a second temporal subsample and a random split. The temporal settings and share the original chronological cutoff and differ in within-pool sampling; uses a random split of the full filtered pool. Each setting uses the same seven listing features and filtering criteria, with 54,000 searches for fitting, 6,000 for validation, and 20,000 for testing.
The results below report mean test NLL (standard deviation) over 10 initialization restarts. Lower is better; the lowest mean NLL in each setting is shown in bold.
| Setting | MNL | LC-MNL | DeepMNL | RUMnet | DeepSets | DeepHalo | TCNet | ||
|---|---|---|---|---|---|---|---|---|---|
| Selected parameters | 7 | 64 | 4.7k | 4.5k | 3.7k | 51k–201k | 20k–37k | 119 | 503 |
| Expedia | 2.6564 (0.0003) | 2.6155 (0.0006) | 3.5200 (0.8472) | 2.7064 (0.0313) | 2.9113 (0.2282) | 2.7734 (0.0585) | 2.7114 (0.0595) | 2.6009 (0.0026) | 2.5983 (0.0018) |
| Expedia | 2.6679 (0.0006) | 2.6232 (0.0011) | 2.9056 (0.2343) | 2.6910 (0.0487) | 2.9174 (0.4163) | 2.7697 (0.0844) | 2.7289 (0.0581) | 2.6090 (0.0034) | 2.6035 (0.0025) |
| Expedia | 2.6744 (0.0002) | 2.6253 (0.0004) | 2.6084 (0.0015) | 2.6010 (0.0012) | 2.5911 (0.0018) | 2.5890 (0.0016) | 2.5818 (0.0036) | 2.6077 (0.0021) | 2.6004 (0.0009) |
On both temporal settings, achieves the lowest mean test NLL and ranks second, while all five deep baselines have higher mean NLL than MNL. This repeated pattern supports the consistency of the results across subsamples at the evaluated temporal boundary.
(2). Why the ranking differs from prior work.
An important difference is the evaluation protocol. TCNet uses random training/validation/test splits [1, Section 6.1], and RUMnet reports ten-fold cross-validation [2, Section 6.2], whereas our original Expedia evaluation separates training and test searches chronologically.
Under the added random split, all five deep baselines outperform MNL, and TCNet achieves the lowest mean test NLL, recovering the qualitative finding highlighted in prior work. Together with the temporal results, this comparison shows that model rankings vary across evaluation settings.
We apologize that reporting only one temporal evaluation in the original manuscript did not make this distinction sufficiently clear. We hope this more comprehensive comparison helps resolve the apparent inconsistency.
(3/3) Response to Reviewer 5KAN
(3). Complementary gains from combining deep encoders with CC.
The comparison above evaluates standalone CC against complete deep choice models. With only 503 parameters, remains competitive: it outperforms both classical baselines in all three Expedia settings and achieves a slightly lower mean NLL than RUMnet on the random split. The strongest model on that split, TCNet, uses 37,217 parameters, approximately 74 times as many.
Our encoder–CC comparison also directly addresses your earlier comment on the relationship between representation learning and choice modeling. To examine their complementarity, we retain the encoder architecture, replace its linear MNL readout with a CC, and train the encoder and circuit jointly. For reference, we reproduce the revised Table 4 below (Section 5.2, p. 10, lines 518–528), reporting mean test NLL (standard deviation) over 10 restarts. Encoder architectures and learning rates are held fixed across the two heads.
| Encoder | Expedia : native head | Expedia : CC head | Trivago: native head | Trivago: CC head |
|---|---|---|---|---|
| None (identity) | 2.6574 (0.0008) | 2.5983 (0.0018) | 2.6400 (0.0006) | 2.4331 (0.0023) |
| MLP (16) | 2.6798 (0.0398) | 2.6469 (0.0152) | 2.4809 (0.0007) | 2.4185 (0.0027) |
| MLP (64, 64) | 2.7006 (0.0401) | 2.6621 (0.0181) | 2.4751 (0.0010) | 2.4148 (0.0016) |
| DeepSets | 2.9113 (0.2282) | 2.8957 (0.1787) | 2.4179 (0.0032) | 2.4114 (0.0118) |
| DeepHalo | 2.8272 (0.1744) | 2.6351 (0.0117) | 2.4151 (0.0027) | 2.4133 (0.0024) |
| TCNet | 2.7377 (0.0686) | 2.6611 (0.0322) | 2.4093 (0.0031) | 2.4013 (0.0027) |
The CC head lowers mean test NLL for every tested encoder on both datasets. In particular, it improves the strongest native encoder on Trivago, TCNet, from 2.4093 to 2.4013, and improves DeepHalo on Expedia from 2.8272 to 2.6351. These results support the complementary roles of learned representations and consideration structure.
⇒ In the revised manuscript, the expanded Expedia comparisons appear in Section 5.1 and Table 3 (pp. 9–10, lines 462–495), with the split definitions and training details in Appendix C.1–C.4 (pp. 33–34). The encoder–CC comparison appears in Section 5.2 and Table 4 (p. 10, lines 498–528).
Summary
We sincerely thank you again for your thoughtful comments. We hope these clarifications and additional comparisons address your concerns and provide a more complete basis for assessing our contribution. If you find the revised manuscript and additional evidence more convincing, we would be grateful if you would reconsider your assessment.
References
[1] Hanzhao Wang, Xiaocheng Li, and Kalyan Talluri. Transformer Choice Net: A Transformer Neural Network for Choice Prediction. arXiv:2310.08716, 2023.
[2] Ali Aouad and Antoine Désir. Representing Random Utility Choice Models with Neural Networks. arXiv:2207.12877, 2022.
(1/3) Response to Reviewer 5KAN
We sincerely thank you for your careful review and constructive feedback, and for recognizing the simplicity of Consideration Circuits and the theoretical characterization of their representational power.
We appreciate your thoughtful questions. Our revisions, guided by your comments, have helped us clarify the scope of our theoretical results and better explain how empirical performance varies across evaluation settings. Specifically, we have clarified the complementary roles of feature representation and consideration structure and expanded the Expedia evaluation to include a random split and an additional subsample using the original temporal cutoff.
Below, we address each of your comments in detail.
1. Complementary Roles of Representation Learning and Decision Structure
Question: As a standalone model, the circuit is limited by the convex hull and translation constraints, whereas transformer-based choice models transform item features non-linearly and are not subject to these limits; the theoretical results compare circuits only with MNL, random-utility models, and circuits of other depths, not with these architectures.
Response: We agree that standalone CC on fixed raw features is subject to these geometric limitations. We respectfully argue that this observation also highlights an opportunity to use CC as an output head within existing deep choice architectures.
Choice heads and deep encoders, including transformers, address complementary modeling dimensions. Deep encoders learn flexible item representations, potentially incorporating menu context; choice heads map these representations to choice probabilities. CC introduce a multistage consideration process into this mapping.
\xrightarrow{\text{Encoder}} \text{Learned representations} \xrightarrow{\text{CC/MNL/other choice head}} \text{Choice probabilities}.$$ Therefore, in our manuscript, we examine CC in two roles: as a standalone choice model on fixed representations and as an output head integrated with a deep encoder. **First, CC as a standalone choice model.** Our theoretical comparisons characterize the additional expressivity obtained from composing MNL units with the feature representation held fixed. Our analysis does not establish a general expressivity ordering between CC and transformer-based choice models; we leave a systematic comparison for future work. Empirically, the standalone results also show that a compact decision structure can be useful even on the original features. With only 503 parameters, $Tree_{63}$ achieves the lowest mean test NLL on the two Expedia temporal splits reported below. On Trivago, its NLL of 2.4331 improves on MNL, LC-MNL, DeepMNL, and RUMnet; for example, RUMnet has NLL 2.4432 with 4,510 parameters. TCNet achieves the best standalone Trivago result, 2.4095, with 37,217 parameters, approximately 74 times as many as $Tree_{63}$. **Second, CC as an output head for learned representations.** As discussed in Section 5.2, CC naturally integrate with existing deep choice architectures: we retain the encoder, replace its linear MNL readout with a CC, and train the encoder and circuit jointly. The encoder learns the representation, while the circuit models successive rounds of consideration on that representation; the geometric restrictions then apply in the learned representation space. Table 4 demonstrates the benefit of this combination: the CC head lowers mean test NLL for every tested encoder on both Expedia and Trivago. Table 4 shows that the CC head lowers mean test NLL for every tested encoder on both Expedia and Trivago. We reproduce the full comparison in our (Response 2.3 in 3/3) below. These results demonstrate the practical value of modeling decision structure with a small parameter budget, alongside the gains from combining CC with learned representations. ⇒ In the revised manuscript, we have clarified this contribution in the Introduction (pp. 2–3, lines 88–110). The two empirical roles are presented in Sections 5.1–5.2 and Tables 3–4 (pp. 9–10), and the encoder–CC composition is illustrated in Figure 4 (Appendix C.7, p. 36). We have also added a numerical illustration of depth–norm separation on the compromise task in Appendix C.9 and Figure 5 (p. 38, lines 1998–2036).