acceptodds
Under review as a conference paper at ICLR 2027

Consideration Circuits: Depth Separation and Universality Beyond a Single Softmax

10 views

Abstract

Most feature-based choice models, classical and deep, score items and apply a single softmax. We introduce consideration circuits (CC), feature-based models of multi-stage choice defined by directed acyclic graphs of multinomial logit (MNL) units. Source units assign probabilities to menu items, and internal units combine predecessor distributions using MNL weights computed from their probability-weighted feature summaries. On a three-item compromise task with fixed non-collinear features, menu-independent random-utility models (RUM), including a single MNL unit, suffer an error bounded away from zero. For CC, in contrast, we establish a sharp depth–norm separation: increasing depth from to reduces the optimal maximum taste-vector norm for error from to . The depth- lower bound holds for arbitrary width and menu-independent routing biases, while a five-node depth- circuit with zero routing biases attains the logarithmic rate. More generally, we characterize two geometric conditions that are necessary and sufficient for approximating arbitrary deterministic choice tables on finite menu families. Under these conditions, depth suffices, while depth achieves optimal logarithmic norm scaling whenever the family contains a non-singleton menu. In experiments, standalone tree circuits with fewer than parameters attain the lowest mean test negative log-likelihood (NLL) among the evaluated models on four fixed-pool benchmarks and the Expedia temporal split. As output heads, CC generalize the linear MNL readout and lower mean test NLL for every tested encoder on Expedia and Trivago.

Primary area:
learning theory
Keywords:
discrete choice model, multinomial logit, softmax, depth separation, universal approximation, context-dependent choice model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_5KANOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies feature-based choice models, and proposes a new method called Consideration circuits. This is defined in terms of a multinomial logits (MNL) structured and applied at nodes of a directed cyclic graph. The paper shows that the representational capacity of this method exceeds a single MNL, and it can represent optimal choice for decoy effect and compromise task examples. They also provide more general representational capacity results for deterministic case, under simplifying assumptions/restrictions on convex combinations and translation. Empirically, they show gains against MNL and when use the proposed method on top of (or as the last layer of) other baselines.

Critical strengths

The proposed method is simple and the paper provides theory characterizing its representational power.

Critical weaknesses

The reviewer is skeptical about the long-term impact of the contributions. As a standalone model, the circuit is limited by the convex hull and translation constraints, whereas transformer-based choice models transform item features non-linearly and are not subject to these limits; the theoretical results compare circuits only with MNL, random-utility models, and circuits of other depths, not with these architectures. The paper’s remaining argument is that the circuit’s constrained structure gives better robustness, but this rests mainly on one dataset (Expedia), in a reduced setting.

Questions for authors

Can the authors justify why on Expedia, the paper’s ranking of models is the reverse of prior work on the same dataset (e.g., the TCNet and RUMnet papers), where deep models clearly outperform MNL, whereas here they all perform worse than MNL?

5 comments
AuthorsOpenReview ↗

(1/3) Response to Reviewer 5KAN

We sincerely thank you for your careful review and constructive feedback, and for recognizing the simplicity of Consideration Circuits and the theoretical characterization of their representational power.

We appreciate your thoughtful questions. Our revisions, guided by your comments, have helped us clarify the scope of our theoretical results and better explain how empirical performance varies across evaluation settings. Specifically, we have clarified the complementary roles of feature representation and consideration structure and expanded the Expedia evaluation to include a random split and an additional subsample using the original temporal cutoff.

Below, we address each of your comments in detail.

 

1. Complementary Roles of Representation Learning and Decision Structure

Question: As a standalone model, the circuit is limited by the convex hull and translation constraints, whereas transformer-based choice models transform item features non-linearly and are not subject to these limits; the theoretical results compare circuits only with MNL, random-utility models, and circuits of other depths, not with these architectures.

Response: We agree that standalone CC on fixed raw features is subject to these geometric limitations. We respectfully argue that this observation also highlights an opportunity to use CC as an output head within existing deep choice architectures.

Choice heads and deep encoders, including transformers, address complementary modeling dimensions. Deep encoders learn flexible item representations, potentially incorporating menu context; choice heads map these representations to choice probabilities. CC introduce a multistage consideration process into this mapping.

\xrightarrow{\text{Encoder}} \text{Learned representations} \xrightarrow{\text{CC/MNL/other choice head}} \text{Choice probabilities}.$$ Therefore, in our manuscript, we examine CC in two roles: as a standalone choice model on fixed representations and as an output head integrated with a deep encoder. **First, CC as a standalone choice model.** Our theoretical comparisons characterize the additional expressivity obtained from composing MNL units with the feature representation held fixed. Our analysis does not establish a general expressivity ordering between CC and transformer-based choice models; we leave a systematic comparison for future work. Empirically, the standalone results also show that a compact decision structure can be useful even on the original features. With only 503 parameters, $Tree_{63}$ achieves the lowest mean test NLL on the two Expedia temporal splits reported below. On Trivago, its NLL of 2.4331 improves on MNL, LC-MNL, DeepMNL, and RUMnet; for example, RUMnet has NLL 2.4432 with 4,510 parameters. TCNet achieves the best standalone Trivago result, 2.4095, with 37,217 parameters, approximately 74 times as many as $Tree_{63}$. **Second, CC as an output head for learned representations.** As discussed in Section 5.2, CC naturally integrate with existing deep choice architectures: we retain the encoder, replace its linear MNL readout with a CC, and train the encoder and circuit jointly. The encoder learns the representation, while the circuit models successive rounds of consideration on that representation; the geometric restrictions then apply in the learned representation space. Table 4 demonstrates the benefit of this combination: the CC head lowers mean test NLL for every tested encoder on both Expedia and Trivago. Table 4 shows that the CC head lowers mean test NLL for every tested encoder on both Expedia and Trivago. We reproduce the full comparison in our (Response 2.3 in 3/3) below. These results demonstrate the practical value of modeling decision structure with a small parameter budget, alongside the gains from combining CC with learned representations. ⇒ In the revised manuscript, we have clarified this contribution in the Introduction (pp. 2–3, lines 88–110). The two empirical roles are presented in Sections 5.1–5.2 and Tables 3–4 (pp. 9–10), and the encoder–CC composition is illustrated in Figure 4 (Appendix C.7, p. 36). We have also added a numerical illustration of depth–norm separation on the compromise task in Appendix C.9 and Figure 5 (p. 38, lines 1998–2036).
AuthorsOpenReview ↗

(2/3) Response to Reviewer 5KAN

2. Expedia Evaluation and Model Rankings

Question: (1) The evidence for robustness rests mainly on one dataset (Expedia), in a reduced setting. (2) Why do deep models perform worse than MNL in the reported Expedia evaluation, whereas prior work, including the TCNet and RUMnet papers, finds that they outperform MNL?

Response: Thank you for pointing out the limitations and lack of clarity in our original evaluation. Addressing these questions has helped us make the comparison more comprehensive and clarify how empirical performance varies across evaluation settings. These concerns primarily relate to Table 3 of the original manuscript, which reported only one temporal evaluation on Expedia. We address them together below.

(1). Expanded Expedia evaluation.

In the revised Table 3 (Section 5.1, p. 9, lines 469–481), we have added a second temporal subsample and a random split. The temporal settings T1T_1 and T2T_2 share the original chronological cutoff and differ in within-pool sampling; RR uses a random split of the full filtered pool. Each setting uses the same seven listing features and filtering criteria, with 54,000 searches for fitting, 6,000 for validation, and 20,000 for testing.

The results below report mean test NLL (standard deviation) over 10 initialization restarts. Lower is better; the lowest mean NLL in each setting is shown in bold.

SettingMNLLC-MNLDeepMNLRUMnetDeepSetsDeepHaloTCNetTree15\mathrm{Tree}_{15}Tree63\mathrm{Tree}_{63}
Selected parameters7644.7k4.5k3.7k51k–201k20k–37k119503
Expedia T1T_12.6564 (0.0003)2.6155 (0.0006)3.5200 (0.8472)2.7064 (0.0313)2.9113 (0.2282)2.7734 (0.0585)2.7114 (0.0595)2.6009 (0.0026)2.5983 (0.0018)
Expedia T2T_22.6679 (0.0006)2.6232 (0.0011)2.9056 (0.2343)2.6910 (0.0487)2.9174 (0.4163)2.7697 (0.0844)2.7289 (0.0581)2.6090 (0.0034)2.6035 (0.0025)
Expedia RR2.6744 (0.0002)2.6253 (0.0004)2.6084 (0.0015)2.6010 (0.0012)2.5911 (0.0018)2.5890 (0.0016)2.5818 (0.0036)2.6077 (0.0021)2.6004 (0.0009)

On both temporal settings, Tree63\mathrm{Tree}_{63} achieves the lowest mean test NLL and Tree15\mathrm{Tree}_{15} ranks second, while all five deep baselines have higher mean NLL than MNL. This repeated pattern supports the consistency of the results across subsamples at the evaluated temporal boundary.

(2). Why the ranking differs from prior work.

An important difference is the evaluation protocol. TCNet uses random training/validation/test splits [1, Section 6.1], and RUMnet reports ten-fold cross-validation [2, Section 6.2], whereas our original Expedia evaluation separates training and test searches chronologically.

Under the added random split, all five deep baselines outperform MNL, and TCNet achieves the lowest mean test NLL, recovering the qualitative finding highlighted in prior work. Together with the temporal results, this comparison shows that model rankings vary across evaluation settings.

We apologize that reporting only one temporal evaluation in the original manuscript did not make this distinction sufficiently clear. We hope this more comprehensive comparison helps resolve the apparent inconsistency.

AuthorsOpenReview ↗

(3/3) Response to Reviewer 5KAN

(3). Complementary gains from combining deep encoders with CC.

The comparison above evaluates standalone CC against complete deep choice models. With only 503 parameters, Tree63Tree_{63} remains competitive: it outperforms both classical baselines in all three Expedia settings and achieves a slightly lower mean NLL than RUMnet on the random split. The strongest model on that split, TCNet, uses 37,217 parameters, approximately 74 times as many.

Our encoder–CC comparison also directly addresses your earlier comment on the relationship between representation learning and choice modeling. To examine their complementarity, we retain the encoder architecture, replace its linear MNL readout with a CC, and train the encoder and circuit jointly. For reference, we reproduce the revised Table 4 below (Section 5.2, p. 10, lines 518–528), reporting mean test NLL (standard deviation) over 10 restarts. Encoder architectures and learning rates are held fixed across the two heads.

EncoderExpedia T1T_1: native headExpedia T1T_1: CC headTrivago: native headTrivago: CC head
None (identity)2.6574 (0.0008)2.5983 (0.0018)2.6400 (0.0006)2.4331 (0.0023)
MLP (16)2.6798 (0.0398)2.6469 (0.0152)2.4809 (0.0007)2.4185 (0.0027)
MLP (64, 64)2.7006 (0.0401)2.6621 (0.0181)2.4751 (0.0010)2.4148 (0.0016)
DeepSets2.9113 (0.2282)2.8957 (0.1787)2.4179 (0.0032)2.4114 (0.0118)
DeepHalo2.8272 (0.1744)2.6351 (0.0117)2.4151 (0.0027)2.4133 (0.0024)
TCNet2.7377 (0.0686)2.6611 (0.0322)2.4093 (0.0031)2.4013 (0.0027)

The CC head lowers mean test NLL for every tested encoder on both datasets. In particular, it improves the strongest native encoder on Trivago, TCNet, from 2.4093 to 2.4013, and improves DeepHalo on Expedia from 2.8272 to 2.6351. These results support the complementary roles of learned representations and consideration structure.

⇒ In the revised manuscript, the expanded Expedia comparisons appear in Section 5.1 and Table 3 (pp. 9–10, lines 462–495), with the split definitions and training details in Appendix C.1–C.4 (pp. 33–34). The encoder–CC comparison appears in Section 5.2 and Table 4 (p. 10, lines 498–528).

 


Summary

We sincerely thank you again for your thoughtful comments. We hope these clarifications and additional comparisons address your concerns and provide a more complete basis for assessing our contribution. If you find the revised manuscript and additional evidence more convincing, we would be grateful if you would reconsider your assessment.

 


References

[1] Hanzhao Wang, Xiaocheng Li, and Kalyan Talluri. Transformer Choice Net: A Transformer Neural Network for Choice Prediction. arXiv:2310.08716, 2023.

[2] Ali Aouad and Antoine Désir. Representing Random Utility Choice Models with Neural Networks. arXiv:2207.12877, 2022.

Reviewer_BmVWOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper targets the multinomial logit (MNL), the single softmax over item scores that most choice models use to predict which item is chosen from a menu. It proposes consideration circuits, which are directed acyclic graphs of MNL units in which source units choose among the menu items, and each interior unit chooses among its predecessors by scoring their probability-weighted feature summaries and mixes their choice distributions accordingly. A single node recovers MNL, a depth-2 circuit is a differentiable consider-then-choose model, and even two sources can reproduce the decoy effect. They provide theoretical results covering depth separation and characterizing when circuits can approximate any deterministic choice table (depth 3 suffices, depth 4 attains logarithmic norm). Experiments compare small tree circuits with classical and deep choice models on four standard benchmarks and two hotel datasets split by time, and use the circuits as output heads for deep encoders.

Critical strengths
  1. A simple, well-motivated generalization of MNL. The model is built from standard MNL units, contains MNL as a special case, and replaces the linear MNL readout of deep choice models without any strong negatives. It is grounded in the consider-then-choose mechanism and in documented context effects (decoy, compromise) that a single softmax cannot produce.
  2. Theoretical results. The theoretical results provide a more complete picture and backing for the method.
Critical weaknesses
  1. The theory is not fully connected with how the model is used.
  • It covers the deterministic tables while training fits stochastic choice probabilities by NLL, and the hotel test NLLs are far from deterministic choice.
  • The cost measure is not motivated. Why the maximum taste-vector norm matters (sharpness, optimization, generalization) is never argued, and the norms of trained circuits are never reported.
  • They are missing experiments that tests the theory, e.g. fitting depth-2 and depth-3 circuits on a synthetic compromise task and reporting error against learned norm.
  1. The empirical claims are stronger than the evidence.
  • Robustness under temporal shift on Expedia. The deep baselines that fall below MNL have large variance (e.g. DeepMNL 3.08±0.553.08 \pm 0.55).
  • Several “best” results are within one standard deviation: SFwork 0.8209±.00140.8209 \pm .0014 vs. LowRankCDM 0.8211±.00150.8211 \pm .0015. Claims such as “lowest on all four” and “for every encoder” should be qualified or backed by significance tests.
Questions for authors

See weaknesses, especially:

  1. Do the universality results extend to stochastic choice tables? Alternatively, is there evidence (theory or synthetic experiments) that depth matters when the targets are stochastic?
  2. What are the taste-vector norms of the trained circuits at each depth in Table 1? Do the depth-3 gains come with smaller norms, as Theorem 4.1 would suggest?
  3. Why are the taste-vector norms important, and why should readers care? This is the main motivation for the theoretical results and it should be clear to the reader.
  4. More ablations on the design are needed. For example, is using a menu independent query ww better than a menu-dependent g(X)g(X) as in MoE?
  5. How are the baselines tuned? What are their tuning grids, training/validation/test splits, etc.
  6. How is the topology of your method selected? From Table 1 it appears, it is selected on test NLL, could you clarify this?
  7. Can you provide significance tests for your results? Especially paired tests with the baselines given some results are within one standard deviation.

Clarifications/suggestions:

  • Presentation: define random-utility models, regularity, and decoy effect where they are first used. You should motivate first what is the desired behavior, why a single softmax and random-utility models cannot produce it, and lastly how circuits do.
  • Presentation: explain the 2−22-\sqrt{2} and 1/21/2 floors and conditions (E)/(T) with a short example.
  • Presentation: describe the baselines of Tables 2–3 in the main text.
  • The decoy mechanism could be stated in one line from the fitted parameters: the decoy takes probability in source A, which raises the price in its summary x(A)x(A); the sink’s taste penalizes price, so its attention moves to B.
  • How should the graph be chosen in practice? For depths 3–5 in Table 1, how are the sources split among parents?
7 comments
AuthorsOpenReview ↗

(1/5) Response to Reviewer BmVW

We sincerely thank you for your thoughtful review and for recognizing the simplicity of CC and the value of our theoretical results. Your feedback has helped us strengthen the connection between theory and practice and improve the empirical evidence and presentation.

Our main revisions and additional analyses include:

  • Universality for strictly positive stochastic choice tables and a clearer motivation for the taste-norm budget.
  • A synthetic depth–norm experiment and measurements of trained taste norms.
  • Expanded Expedia evaluations, significance tests, and qualified empirical claims.
  • Clarifications of baseline tuning, routing, topology selection, and presentation.

We address all of your comments below, grouping related concerns and indicating their original labels.

 

1. Universality and Depth for Stochastic Choice (W1.1, Q1)

Question: The theoretical results concern deterministic choice tables, whereas NLL training fits stochastic choice probabilities. Do the universality results extend to stochastic choice tables? Alternatively, is there theoretical or synthetic evidence that depth matters when the targets are stochastic?

Response: Thank you for highlighting this important connection to stochastic choice learning, which we had not developed in the original manuscript. Our existing constructions provide a natural starting point: by extending them and calibrating the readout weights, we obtain exact representations of strictly positive stochastic choice tables under the same admissibility conditions. We have added this result as Theorem 4.5, with proof in Appendix B.9.

Theorem 4.5 (Stochastic universality). Fix a nonempty menu family F⊆2[N]∖{∅}\mathcal F\subseteq 2^{[N]}\setminus\{\varnothing\}, and write

\qquad D=\sum_{S\in\mathcal F}(|S|-1).$$ There exist two fixed topologies $G_{\mathcal F}^{(3)}$ and $G_{\mathcal F}^{(4)}$, depending only on $\mathcal F$ and independently of the features and target table, with the following properties. For every catalog $X=(x_1,\ldots,x_N)$ satisfying the same admissibility conditions (E) and (T), and every strictly positive stochastic choice table $p=(p^S)_{S\in\mathcal F}$: 1. **Exact representation at depth 3.** $G_{\mathcal F}^{(3)}$ has depth at most $3$ and at most $2D+2$ nodes. There exist finite parameters $\theta^{(3)}$ such that $$P_{G_{\mathcal F}^{(3)},\theta^{(3)}}(\cdot\mid S)=p^S \qquad(S\in\mathcal F).$$ 2. **Exact representation with logarithmic norm at depth 4.** $G_{\mathcal F}^{(4)}$ has depth at most $4$ and at most $(m+1)D+3$ nodes. Writing $\alpha=\min_{S\in\mathcal F,\,i\in S}p_i^S>0$, there exist finite parameters $\theta^{(4)}$ such that $$P_{G_{\mathcal F}^{(4)},\theta^{(4)}}(\cdot\mid S)=p^S \qquad(S\in\mathcal F),$$ and $$\max_v\lvert w_v^{(4)}\rvert_2 \le C_{X,\mathcal F}\log(1/\alpha),$$ where $C_{X,\mathcal F}<\infty$ is independent of the target table. As in the deterministic case, the same admissibility conditions suffice at depth 3 for universality and at depth 4 for a logarithmic taste-norm upper bound. The key difference is that strictly positive stochastic targets admit exact representation with finite parameters, with the depth-4 bound scaling as $O(\log(1/\alpha))$, whereas deterministic targets are approximated to error $\varepsilon$ with a bound of $O(\log(1/\varepsilon))$. &nbsp; ## 2. Motivation for the Taste-Norm Budget (W1.2, Q3) **Question:** *Why is the maximum taste-vector norm a meaningful resource for the theoretical analysis, and why should readers care about it? What is its relevance to choice sharpness, optimization, or generalization?* **Response:** Thank you for highlighting the need to motivate this resource. Norm constraints are an established way to study representational complexity and depth separation, including when network width is unrestricted [2, 3]. For CC, we have added a choice-theoretic interpretation of this budget based on rational inattention in Section 4.1. Under Shannon information costs, its standard-logit case yields choice probabilities proportional to $\exp(u_i/\lambda)$, where $\lambda$ is the unit cost of information [1]. For linear underlying utilities $u_i=\bar w^\top x_i$, the effective taste vector is therefore $$w=\frac{\bar w}{\lambda}, \qquad \|w\|_2=\frac{\|\bar w\|_2}{\lambda}.$$ With the feature and underlying utility scales held fixed, taste norm is thus proportional to the inverse unit information cost, motivating its interpretation as a budget for local choice precision, or sensitivity to feature differences.
AuthorsOpenReview ↗

(2/5) Response to Reviewer BmVW

In CC, each source is an MNL chooser over items, and each interior node is an MNL chooser over the feature summaries provided by its predecessors. The budget max⁡v∥wv∥2\max_v\|w_v\|_2 consequently bounds the sensitivity available at every local decision stage. Our depth-separation question is then whether additional stages can realize the same contextual choice behavior with a lower maximum local precision requirement.

References

[1] Filip Matějka and Alisdair McKay. Rational Inattention to Discrete Choices: A New Foundation for the Multinomial Logit Model. American Economic Review, 105(1):272–298, 2015.

[2] Greg Ongie, Rebecca Willett, Daniel Soudry, and Nathan Srebro. A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Case. ICLR, 2020.

[3] Suzanna Parkinson, Greg Ongie, Rebecca Willett, Ohad Shamir, and Nathan Srebro. Depth Separation in Norm-Bounded Infinite-Width Neural Networks. COLT, PMLR 247:4082–4114, 2024.

 

3. Empirical Evidence for the Depth–Norm Tradeoff (W1.2, W1.3, Q2)

Question: Can the depth–norm prediction be illustrated by training depth-2 and depth-3 circuits on the synthetic compromise task and reporting error against learned norm? What are the taste-vector norms of the trained circuits at each depth in Table 1, and do the depth-3 gains come with smaller norms?

Response: Thank you for this suggestion. We had intended to examine this connection empirically, but found that relating trained norms to the theoretical optimum is not trivial. Theorem 4.1 compares the minimum maximum-node norm required to achieve the same target error on the compromise task. Our hotel experiments minimize NLL without a norm penalty or constraint and reach different NLL levels, so their learned norms do not estimate this matched-error optimum.

To examine this relationship more directly, we have added a synthetic experiment on the compromise task in Appendix C.9 (Figure 5). We evaluate the explicit constructions and train depth-2 stars with 2, 4, or 16 sources and the five-node depth-3 circuit, using 10 random initializations per architecture. Training minimizes the worst-menu NLL without a norm penalty or constraint, and we plot the achieved error against the maximum node taste norm.

The constructions and the width-independent depth-2 lower bound illustrate the norm separation predicted by Theorem 4.1. Random training also finds low-norm, high-accuracy depth-3 solutions: two of the ten runs enter a region excluded by the theoretical lower bound for every depth-2 circuit, regardless of width or menu-independent edge biases. These results numerically illustrate the predicted advantage of additional depth. We report all initializations and clarify that unconstrained training does not consistently recover this norm advantage across seeds.

We focus the supplementary norm analysis on depths 2 and 3, which are the depths compared in Theorem 4.1. We also conducted five supplementary training restarts for each of the depth-2 and depth-3 tree configurations in Table 1 on Expedia and Trivago. Both architectures have 63 nodes and 503 parameters. At the validation-selected checkpoints, we measured the maximum node taste norm, B=max⁡v∥wv∥2B=\max_v\|w_v\|_2, excluding edge biases. The following results report mean ± sample standard deviation.

DatasetDepthTest NLLMaximum taste norm BB
Expedia22.6102 ± 0.001011.3160 ± 4.7106
Expedia32.5976 ± 0.001211.9822 ± 1.9174
Trivago22.4539 ± 0.000417.7442 ± 0.9595
Trivago32.4382 ± 0.001420.6942 ± 4.2637

The mean learned norm does not decrease at depth 3 on either dataset, although depth 3 achieves lower test NLL on both.

One possible interpretation is that, under unconstrained training, the additional expressive flexibility afforded by depth is used to attain lower NLL rather than smaller learned norms. This is compatible with the theoretical norm–accuracy tradeoff, although these experiments do not directly establish an optimization guarantee.

 

4. Baseline Evaluation and Robustness Under Temporal Shift (W2.1, Q5)

Question: How well is the claimed robustness to temporal shift on Expedia supported, given the large variability of the deep baselines? How are the baselines tuned and evaluated, including their hyperparameter grids, training procedures, and training/validation/test splits?

Response: Thank you for raising this concern and for pointing out the lack of clarity in our original manuscript. Your comments have helped us make the comparison more comprehensive and clarify the basis for our conclusions.

AuthorsOpenReview ↗

(3/5) Response to Reviewer BmVW

To address these points, we have expanded the reported Expedia comparison to 10 initialization restarts per configuration. The revised Table 3 also includes an additional temporal subsample, T2T_2, using the original cutoff, and a random-split comparison, RR. Candidate configurations are selected separately within each setting by mean validation NLL; DeepSets and CC use fixed configurations.

The deep-baseline tuning grids, training protocol, and data splits are detailed in Appendix C.1, C.2, and C.4. We summarize them below.

For the Table 3 comparison, each candidate configuration is evaluated with 10 initialization restarts. Within each dataset setting, we select the configuration with the lowest mean validation NLL and report its test performance over all 10 restarts. DeepSets uses a fixed configuration.

ModelExpedia configurationsTrivago configurations
DeepMNLDepth 2; width ∈{16,64}\in\{16,64\}; learning rate 10−210^{-2}Width 64; depth ∈{2,3}\in\{2,3\}; learning rate 10−310^{-3}
RUMnetK∈{5,10}K\in\{5,10\} heterogeneity samples; learning rate 10−210^{-2}Same as Expedia
DeepSetsFixed width-32 pooled-context encoder with a linear readout; learning rate 10−210^{-2}Same as Expedia
DeepHalo4 heads, 2 layers; embedding dimension ∈{64,128}\in\{64,128\}; learning rate 10−310^{-3}Same as Expedia
TCNetWidth 32, 4 heads; either 1 layer with learning rate 10−310^{-3} or 2 layers with learning rate 10−210^{-2}2 layers, 4 heads; width ∈{32,64}\in\{32,64\}; learning rate 10−210^{-2}

All models use Adam with batch size 256, at most 150 epochs, and validation-based early stopping with patience 15 and a minimum of 30 epochs. The checkpoint with the best validation NLL is restored before evaluation.

The resulting fitting/validation/test sizes are:

SettingFittingValidationTest
Expedia T1T_154,0006,00020,000
Expedia T2T_254,0006,00020,000
Expedia RR54,0006,00020,000
Trivago55,0256,11420,380

Splitting is performed at the search/session level, and all models use the same data partition within each setting:

  • Expedia temporal splits: searches are divided into 75/25 pools chronologically, from which 60,000 training-pool searches and 20,000 test searches are sampled. The latest 10% of the sampled training pool is reserved for validation.
  • Expedia random split: 80,000 searches are sampled and randomly assigned to a 60,000-search training pool and a 20,000-search test set. A random 10% of the training pool is reserved for validation.
  • Trivago: all 81,519 sessions are divided chronologically into 61,139 training-pool sessions and 20,380 test sessions. The latest 10% of the training pool is reserved for validation.

The results below report test NLL as mean ± standard deviation over all 10 restarts; lower is better.

ModelTemporal T1T_1Temporal T2T_2Random RR
MNL2.6564 ± 0.00032.6679 ± 0.00062.6744 ± 0.0002
LC-MNL2.6155 ± 0.00062.6232 ± 0.00112.6253 ± 0.0004
DeepMNL3.5200 ± 0.84722.9056 ± 0.23432.6084 ± 0.0015
RUMnet2.7064 ± 0.03132.6910 ± 0.04872.6010 ± 0.0012
DeepSets2.9113 ± 0.22822.9174 ± 0.41632.5911 ± 0.0018
DeepHalo2.7734 ± 0.05852.7697 ± 0.08442.5890 ± 0.0016
TCNet2.7114 ± 0.05952.7289 ± 0.05812.5818 ± 0.0036
Tree15\mathrm{Tree}_{15}2.6009 ± 0.00262.6090 ± 0.00342.6077 ± 0.0021
Tree63\mathrm{Tree}_{63}2.5983 ± 0.00182.6035 ± 0.00252.6004 ± 0.0009

On both temporal partitions, Tree63Tree_{63} achieves the lowest mean test NLL and Tree15Tree_{15} ranks second, while every deep baseline has higher mean NLL than MNL. On the random split, all five deep baselines outperform MNL, and TCNet achieves the lowest mean NLL overall.

Their strong random-split performance demonstrates that our implementations and tuning procedure can yield competitive deep baselines on Expedia. The contrast across split types supports sensitivity to temporal distribution shift as an explanation for their weaker temporal-test performance, and supports CC's relative robustness in the evaluated temporal settings.

 

5. Statistical Significance and Qualification of Empirical Claims (W2.2, Q7)

Question: Several reported improvements are small relative to the reported standard deviations. Can you provide significance tests, particularly paired comparisons with the baselines, and qualify claims such as “lowest on all four” and “for every encoder” where appropriate?

Response: Thank you for pointing this out. In this revision, we have taken greater care to qualify our claims about performance improvements and to present the results across datasets. In particular, we distinguish improvements in mean NLL from statistically significant gains.

Reviewer_xb42OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes consideration circuits (CC), a choice model built from multinomial logit (MNL) units connected in a directed graph. Source units assign probabilities to items in a menu. Later units combine these distributions using probability-weighted feature summaries. This allows the model to capture context effects that a single MNL cannot.

The main theoretical result is a depth separation on a three-item compromise task. Depth 2 needs a taste-vector norm of order log(1/ε)/ε to reach error ε, while depth 3 only needs order log(1/ε). The authors also give conditions for approximating any deterministic choice table on a finite set of menus. In experiments, small circuits perform well against classical choice models and some deep baselines. Using CC as a readout head also lowers mean NLL for all tested encoders on Expedia and Trivago.

Critical strengths
  • The model is easy to understand, and the paper is clearly written. It builds on familiar MNL units, preserves permutation equivariance, and includes standard MNL as a special case.

  • The theory is clean and solid. The depth-2 lower bound holds even with arbitrary width and routing biases, which makes the depth-3 improvement meaningful. I also appreciate that the universality results state their geometric assumptions and include a counterexample to logarithmic norm scaling at depth 3.

  • The empirical results are encouraging, especially the clear gains on Swissmetro and ModeCanada and the strong performance of the small circuit on Expedia.

Critical weaknesses
  • The paper does not show whether the theoretical advantage of deeper circuits actually leads to better performance on real-world choice data (stochastic vs deterministic).
  • The universality results focus on deterministic choices over fixed menus and can require very large circuits.
  • The temporal robustness claim relies on a single Expedia split, where several deep baselines perform poorly or show high variance.
Questions for authors
  • How do the Expedia results change with a random split or another temporal cutoff?
  • Could you provide statistical significance tests or confidence intervals for the smaller NLL improvements in Tables 2 and 4?
4 comments
AuthorsOpenReview ↗

(1/3) Response to Reviewer xb42

We sincerely thank you for your careful review and constructive feedback, and for recognizing the clarity of our model, the strength of our theoretical results, and the encouraging empirical findings. Your comments have helped us clarify the connection between the theory and empirical results and strengthen the presentation of our evidence.

Below, we address each concern, clarifying the results already reported in the original manuscript and presenting the additional theoretical and experimental evidence.

 

1. Stochastic Universality and the Empirical Benefits of Depth (W1, W2)

Question: The paper does not show whether the theoretical advantage of deeper circuits actually leads to better performance on real-world choice data (stochastic vs deterministic). The universality results focus on deterministic choices over fixed menus and can require very large circuits.

Response: Thank you for highlighting the need to clarify the connection between our theoretical results and stochastic choice prediction. Our existing constructions provide a natural starting point: by extending them and calibrating the readout weights, we obtain exact representations of strictly positive stochastic choice tables under the same admissibility conditions. We have added this result as Theorem 4.5, with proof in Appendix B.9.

Theorem 4.5 (Stochastic universality). Fix a nonempty menu family F⊆2[N]∖{∅}\mathcal F\subseteq 2^{[N]}\setminus\{\varnothing\}, and write

\qquad D=\sum_{S\in\mathcal F}(|S|-1).$$ There exist two fixed topologies $G_{\mathcal F}^{(3)}$ and $G_{\mathcal F}^{(4)}$, depending only on $\mathcal F$ and independently of the features and target table, with the following properties. For every catalog $X=(x_1,\ldots,x_N)$ satisfying the same admissibility conditions (E) and (T), and every strictly positive stochastic choice table $p=(p^S)_{S\in\mathcal F}$: 1. **Exact representation at depth 3.** $G_{\mathcal F}^{(3)}$ has depth at most $3$ and at most $2D+2$ nodes. There exist finite parameters $\theta^{(3)}$ such that $$P_{G_{\mathcal F}^{(3)},\theta^{(3)}}(\cdot\mid S)=p^S \qquad(S\in\mathcal F).$$ 2. **Exact representation with logarithmic norm at depth 4.** $G_{\mathcal F}^{(4)}$ has depth at most $4$ and at most $(m+1)D+3$ nodes. Writing $\alpha=\min_{S\in\mathcal F,\,i\in S}p_i^S>0$, there exist finite parameters $\theta^{(4)}$ such that $$P_{G_{\mathcal F}^{(4)},\theta^{(4)}}(\cdot\mid S)=p^S \qquad(S\in\mathcal F),$$ and $$\max_v\lVert w_v^{(4)} \rVert_2 \le C_{X,\mathcal F}\log(1/\alpha),$$ where $C_{X,\mathcal F}<\infty$ is independent of the target table. As in the deterministic case, the same admissibility conditions suffice at depth 3 for universality and at depth 4 for a logarithmic taste-norm upper bound. The key difference is that strictly positive stochastic targets admit exact representation with finite parameters, with the depth-4 bound scaling as $O(\log(1/\alpha))$, whereas deterministic targets are approximated to error $\varepsilon$ with a bound of $O(\log(1/\varepsilon))$. This extension connects the universality guarantee to stochastic choice distributions, including distributions far from deterministic choice. Regarding the empirical benefits of depth, we also appreciate the opportunity to make the relevance of our original depth comparison clearer. The original manuscript already reported a comparison on Expedia and Trivago with the total number of nodes fixed at 63 (Table 1, p. 7). The tree configurations also share the same parameter count of 503. Increasing depth from 2 to 3 reduces mean test NLL from 2.6101 to 2.5980 on Expedia and from 2.4540 to 2.4389 on Trivago. The corresponding reductions for fully connected circuits are 0.0109 and 0.0160. All entries average 10 restarts. These results show an improvement in real-world stochastic choice prediction from depth 2 to depth 3 at a fixed node budget, complementing the stochastic expressivity guarantee above. Relating these trained models to the theoretical norm optimum requires further care. Theorem 4.1 compares the minimum maximum-node norm required to achieve the same target error on the compromise task. Our hotel experiments minimize NLL without a norm penalty or constraint and reach different NLL levels, so their learned norms do not estimate this matched-error optimum. To examine the norm–accuracy relationship more directly, we have also added a synthetic experiment on the deterministic compromise task in Appendix C.9 (Figure 5). Two of the ten depth-3 training runs enter a region excluded by the theoretical lower bound for every depth-2 circuit, regardless of width or menu-independent edge biases. We report all initializations and clarify that unconstrained training does not consistently recover this norm advantage across seeds.
AuthorsOpenReview ↗

(2/3) Response to Reviewer xb42

Regarding circuit size, we appreciate the opportunity to clarify how the construction scales with the number of observed menus. When specialized to a fixed family of mm observed menus, our original depth-3 construction in Appendix B.5 requires O(m)O(m) nodes. At depth 4, O(m2)O(m^2) nodes suffice for deterministic universality with the optimal logarithmic norm scaling (Theorem 4.4). The exponential upper bound in NN in Theorem 4.2 comes from allocating modules for all possible non-singleton menus so that a single topology works for every menu family. These are worst-case universality constructions; our empirical evaluations use compact fixed circuits.

 

2. Temporal and Random-Split Evaluation on Expedia (W3, Q1)

Question: The temporal robustness claim relies on a single Expedia split, where several deep baselines perform poorly or show high variance. How do the Expedia results change with a random split or another temporal cutoff?

Response: Thank you for pointing out the limitations of the original evaluation. To address this concern, we have expanded the reported Expedia comparison to 10 initialization restarts per configuration and added an additional temporal split and a random-split comparison. The revised Table 3 reports the following test NLLs as mean ± standard deviation over all 10 restarts; lower is better.

ModelTemporal T1T_1Temporal T2T_2Random RR
MNL2.6564 ± 0.00032.6679 ± 0.00062.6744 ± 0.0002
LC-MNL2.6155 ± 0.00062.6232 ± 0.00112.6253 ± 0.0004
DeepMNL3.5200 ± 0.84722.9056 ± 0.23432.6084 ± 0.0015
RUMnet2.7064 ± 0.03132.6910 ± 0.04872.6010 ± 0.0012
DeepSets2.9113 ± 0.22822.9174 ± 0.41632.5911 ± 0.0018
DeepHalo2.7734 ± 0.05852.7697 ± 0.08442.5890 ± 0.0016
TCNet2.7114 ± 0.05952.7289 ± 0.05812.5818 ± 0.0036
Tree15\mathrm{Tree}_{15}2.6009 ± 0.00262.6090 ± 0.00342.6077 ± 0.0021
Tree63\mathrm{Tree}_{63}2.5983 ± 0.00182.6035 ± 0.00252.6004 ± 0.0009

On both temporal splits, Tree63Tree_{63} achieves the lowest mean test NLL and Tree15Tree_{15} ranks second, while every deep baseline has higher mean NLL than MNL.

On the random split, the deep baselines demonstrate strong predictive performance, with TCNet achieving the best result. Our 503-parameter Tree63Tree_{63} also remains competitive, outperforming both MNL and LC-MNL. These results demonstrate that the deep baselines can learn effective models of Expedia under our training and tuning protocol. The contrast across split types supports sensitivity to temporal distribution shift as an explanation for their weaker temporal-test performance, and supports CC's relative robustness in the evaluated temporal settings.

AuthorsOpenReview ↗

(3/3) Response to Reviewer xb42

3. Statistical Significance and Qualified Empirical Claims (Q2)

Question: Could you provide statistical significance tests or confidence intervals for the smaller NLL improvements in Tables 2 and 4?

Response: Thank you for this suggestion. We have taken greater care in this revision to qualify our empirical claims, distinguishing improvements in mean NLL from statistically significant gains. We provide the corresponding statistical tests below.

Table 2. We have replaced the claim of being “best on all four benchmarks” with the more precise statement that CC are competitive on SFwork and SFshop and achieve the lowest mean test NLL among the evaluated models on Swissmetro and ModeCanada. On the latter two benchmarks, we compare Tree63 with the strongest baseline by mean test NLL:

BenchmarkStrongest baselineBaseline NLLCC NLLNLL reductionHolm-adjusted p
SwissmetroLC-MNL(16)0.7518380.7267060.0251320.00523
ModeCanadaLowRankCDM(1)0.5842970.5590230.0252740.000130

These results use two-sided paired t-tests over 10 restarts, with Holm correction across all 136 baseline–circuit comparisons in Table 2, including the evaluated model sizes. Both improvements remain significant at the 0.05 level after correction. We do not claim statistically significant superiority on SFwork or SFshop.

Table 4. We test the linear readout against the CC readout separately for each encoder and dataset. Because the two readouts use different initialization seeds, we use two-sided Welch t-tests over 10 restarts per configuration, with Holm correction across all 12 comparisons. Here, ΔNLL denotes linear-readout NLL minus CC-readout NLL; positive values favor CC.

EncoderExpedia ΔNLLHolm-adjusted pTrivago ΔNLLHolm-adjusted p
Id0.0591165.43 × 10⁻¹⁸0.2068472.58 × 10⁻²⁰
MLP(16)0.0328320.12850.0624483.02 × 10⁻¹⁴
MLP(64,64)0.0385420.08200.0602676.26 × 10⁻²²
DeepSets0.0155790.86700.0065110.3634
DeepHalo0.1921440.04810.0018350.3634
TCNet0.0766330.04810.0080256.45 × 10⁻⁵

The CC readout lowers mean test NLL in all 12 comparisons, with seven improvements significant after correction. In particular, TCNet already has the lowest mean NLL among the tested native encoders on Trivago, and the CC readout further reduces its NLL from 2.409299 to 2.401274, with a Holm-adjusted p-value of 6.45 × 10⁻⁵. The manuscript's claim of lower mean test NLL concerns averages over restarts; the tests above further show which improvements are statistically significant.

 

We sincerely thank you again for your careful review and constructive suggestions. We have carefully addressed each of your concerns and questions, particularly through the new stochastic universality theorem, the expanded empirical evaluation, and the statistical analyses presented above. We hope these revisions and additional results address your concerns and provide a stronger basis for evaluating our work.

Dear Reviewer xb42,

We would appreciate it if you could take some time to look through our responses, and we hope your concerns have been well addressed.

Thank you again for all the constructive feedback, and we would be very grateful if you could consider revisiting your assessment of our work!

Best,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.