- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This work investigates the internal decision-making mechanisms of three representative autoregressive NCO solvers for routing problems: AM and POMO (HELD) and LEHD. The authors build a multi-level interpretability framework combining geometric trajectory analysis, linear representation probing, and causal activation interventions. The key findings are: 1) HELD rely on a rigid geometric scanning strategy with systematic layer-wise oscillations and fixed rotational bias; 2) LEHD adopts a two-component navigation strategy: local multi-step look-ahead planning and global directional guidance from the start node; 3) LEHD organizes node representations into role-specific canonical latent spaces across instances.
Critical strengths
- This work moves beyond prior purely correlational probing studies and introduces causal activation interventions to the mechanistic analysis of NCO solvers. The tiered pipeline establishes a rigorous methodological template for the field.
- The study systematically contrasts two dominant encoder-decoder paradigms and links architectural capacity allocation to qualitatively distinct solution strategies.
- Core findings are replicated across both TSP and CVRP.
Critical weaknesses
- The core comparison between HELD models (Trained via RL) and LEHD (trained via SL) is fundamentally confounded. There is no way to disentangle whether the observed differences in strategy rigidity and look-ahead capability stem from encoder-decoder allocation or from the training paradigm.
- The work remains entirely descriptive and does not translate its mechanistic findings into concrete design improvements, nor does it include ablation or validation experiments demonstrating that leveraging these insights yields measurable performance gains.
- The claim that start/depot nodes provide global angular guidance is well supported in TSP, but evidence is weak in CVRP. As shown in the reported results, depot patching produces very low LCS values across all angle ranges, as capacity constraints dominate route partitioning.
- The high cross-instance similarity of current-node representations is interpreted as a canonical functional reference frame, but the analysis does not control for inherent geometric similarities of current nodes (e.g., local neighborhood statistics). It remains ambiguous whether the alignment reflects role-specific encoding or simply shared local geometric properties.
- The probe-guided steering experiment only performs activation amplification along the probe direction, with no corresponding inhibition/ablation condition to verify causal necessity. Additionally, only joint steering across horizons 2–5 is tested, without single-horizon ablations.
Questions for authors
See weaknesses.
10 comments
Rebuttal (Part 2/3): Capacity Constraints and Cross-Instance Representation Alignment (W3, W4)
W3: Thank you for your thoughtful observation. As you noted and as we discussed in Appendix G.3, because CVRP tours are constrained by vehicle capacity limits, these constraints affect how the intervention alters the route's overall direction and thus lower the LCS values. Similarly, because each individual CVRP route consists of far fewer nodes than a full TSP tour, this also lowers the Reverse LCS values. To isolate the effect of capacity constraints, we ran this experiment on CVRP-500 instances with unconstrained vehicle capacities (1.05 ) so that the model is not forced to return to the depot multiple times. The results are shown below (and in Table 27 of the revised manuscript). We observe that under unconstrained capacity, forward LCS increases across all angle ranges, and as the donor angle range increases, Rev. LCS rises consistently from 0.144 to 0.321, mirroring the behavior observed in TSP. Interestingly, even when the vehicle capacity is large enough that a single route is optimal, LEHD still returns to the depot an average of 2.58 times (compared to 20.44 under the original capacity), showing how the learned policy interacts with route partitioning.
CVRP-500 Route Rollout-Order Similarity Under Depot-Node Patching (Appendix G.3, Table 27)
C: Original capacity; Unconstr.: Unconstrained capacity. LCS: Forward-order similarity; Rev.: Reverse-order similarity.
| Angle | LEHD-C | LEHD-C Rev. | LEHD-U | LEHD-U Rev. | POMO-C | POMO-C Rev. |
|---|---|---|---|---|---|---|
| 0°–30° | 0.496 | 0.123 | 0.593 | 0.144 | 0.168 | 0.079 |
| 30°–60° | 0.318 | 0.126 | 0.431 | 0.242 | 0.108 | 0.065 |
| 60°–90° | 0.221 | 0.117 | 0.365 | 0.266 | 0.080 | 0.064 |
| 90°–120° | 0.155 | 0.107 | 0.300 | 0.281 | 0.080 | 0.071 |
| 120°–150° | 0.117 | 0.107 | 0.267 | 0.294 | 0.104 | 0.083 |
| 150°–180° | 0.115 | 0.111 | 0.293 | 0.321 | 0.151 | 0.118 |
W4: Please note that in the average cross-instance cosine similarity reported in Table 2 (and Table 28 for CVRP), for each layer we compare representations across independent random problem instances at the same decoding step t and then again averaged across these decoding steps, so the compared current nodes occupy arbitrary coordinates and varied local neighborhoods across instances. Furthermore, the encoder layer (Enc. 1 in Table 2), which processes pure instance and local neighborhood geometry prior to role-conditioned decoding, serves as a direct control: in Enc. 1, the cross-instance similarity of the current node (0.194) is virtually identical to that of random unvisited nodes (0.193) and the start node (0.188), whereas the strong alignment of the current node (0.767–0.886) emerges only in the deeper decoder layers (Dec. 2–5), confirming that the alignment reflects role-specific encoding in the decoder rather than shared local geometric properties.
Rebuttal (Part 3/3): Causal Steering Controls and Horizon Ablations (W5)
W5: Per your suggestion, we have added an inhibition condition for the alternative branch (using a negative =-5 for probe-guided steering). As shown below, the logit gap shift () becomes negative, weakening the preference for the alternative branch as expected. We also steered the main (greedy) branch using both positive and negative (toward and away from the probe direction) and observed the exact opposite trend compared to the alternative branch: when we steer the main branch toward the probe direction (>0), becomes negative (the main branch becomes even more favorable), whereas when we steer it away from the probe direction (<0), becomes positive (shifting preference toward the alternative branch), verifying causal necessity.
We have also conducted single-horizon ablations (h{2,3,4,5} individually), observing the same directional response across both branches and finding that the steering effect is strongest at H2 and gradually diminishes at further horizons (H3,H4,H5). All of these results, along with representation-space off-manifold diagnostics, are presented in Appendix E.3 (Tables 20–24) of the revised manuscript.
Joint H2–H5 Main-Branch Steering on Uniform TSP-100 (Table 22)
| Horizon | Probe-Guided | Random Direction | Random Horizon | |
|---|---|---|---|---|
| H2–H5 | -5 | +0.326 ± 0.047 | -0.046 ± 0.020 | -0.015 ± 0.017 |
| H2–H5 | +5 | -0.396 ± 0.054 | +0.035 ± 0.019 | +0.057 ± 0.018 |
Joint H2–H5 Alternative-Branch Steering on Uniform TSP-100 (Table 24)
| Horizon | Probe-Guided | Random Direction | Random Horizon | |
|---|---|---|---|---|
| H2–H5 | -5 | -0.191 ± 0.042 | +0.045 ± 0.024 | +0.014 ± 0.022 |
| H2–H5 | +5 | +0.430 ± 0.064 | +0.025 ± 0.028 | -0.001 ± 0.029 |
Single-Horizon Main-Branch Steering on Uniform TSP-100 (Table 21)
| Horizon | Probe-Guided | Random Direction | Random Horizon | |
|---|---|---|---|---|
| H2 | -5 | +0.205 ± 0.033 | -0.020 ± 0.013 | +0.010 ± 0.006 |
| H2 | +5 | -0.273 ± 0.047 | +0.032 ± 0.015 | +0.017 ± 0.009 |
| H3 | -5 | +0.079 ± 0.017 | -0.012 ± 0.008 | +0.004 ± 0.006 |
| H3 | +5 | -0.108 ± 0.018 | +0.003 ± 0.008 | +0.017 ± 0.009 |
| H4 | -5 | +0.040 ± 0.011 | -0.003 ± 0.007 | -0.002 ± 0.006 |
| H4 | +5 | -0.079 ± 0.014 | -0.014 ± 0.007 | +0.020 ± 0.009 |
| H5 | -5 | -0.025 ± 0.011 | -0.006 ± 0.011 | +0.002 ± 0.007 |
| H5 | +5 | 0.000 ± 0.013 | +0.001 ± 0.008 | +0.030 ± 0.010 |
Single-Horizon Alternative-Branch Steering on Uniform TSP-100 (Table 23)
| Horizon | Probe-Guided | Random Direction | Random Horizon | |
|---|---|---|---|---|
| H2 | -5 | -0.124 ± 0.026 | +0.043 ± 0.014 | +0.008 ± 0.013 |
| H2 | +5 | +0.299 ± 0.049 | -0.034 ± 0.021 | +0.006 ± 0.012 |
| H3 | -5 | -0.068 ± 0.017 | -0.009 ± 0.010 | +0.014 ± 0.012 |
| H3 | +5 | +0.094 ± 0.018 | +0.015 ± 0.010 | +0.017 ± 0.011 |
| H4 | -5 | -0.012 ± 0.013 | -0.008 ± 0.008 | +0.022 ± 0.011 |
| H4 | +5 | +0.059 ± 0.016 | +0.032 ± 0.008 | +0.012 ± 0.011 |
| H5 | -5 | +0.031 ± 0.014 | +0.015 ± 0.010 | -0.008 ± 0.012 |
| H5 | +5 | -0.011 ± 0.018 | +0.007 ± 0.010 | -0.002 ± 0.010 |
Rebuttal (Part 1/3): Experimenting on Different Training Paradigms and Design Improvements (W1, W2)
Dear Reviewer Lr1J,
Thank you for your detailed feedback. We appreciate that you found our experimental framework to be a rigorous methodological template for this field, as well as your recognition of the breadth of our experiments across different architectures and routing problems. Below, we address each of your concerns. We have also uploaded the revised manuscript incorporating these additions, along with references to the updated sections:
W1: Thank you for pointing this out. Since both the training paradigm and the encoder-decoder configuration can influence the final decision-making mechanism, the most direct way to disentangle their respective roles is to evaluate cross-combinations of architectures and training paradigms—specifically, POMO trained with supervised learning (
POMO-SL) and a light-encoder, heavy-decoder architecture trained outside standard SL. As explained in detail in our response to W2 and Q1 of Reviewer 3J3R, due to the substantial computational cost of trainingLEHD-RLto full convergence within the rebuttal window, we evaluatedPOMO-SLalongside SIL-SIT [1], a heavy-decoder model trained via self-improved reconstruction. Across all of our mechanistic evaluations—future-action linear probes (Figure 4b), start-node patching probability drop (Figure 5b), LCS and Rev. LCS rollout reversal (Table 1), and onion-layer oscillations (Figure 3a)—POMO-SLbehaves just like standard POMO (except that its unidirectional angular bias is eliminated), whereas SIL-SIT behaves just like LEHD. This demonstrates that multi-step look-ahead, start-node global directional guidance, and layer-oscillation behavior stem from the encoder-decoder architectural allocation rather than the training paradigm, while only the rotational sweeping bias stems from RL training. We are also runningLEHD-RLand are committed to including it in the final manuscript. For the full quantitative tables across all benchmarks, please see our response to W2 and Q1 of Reviewer 3J3R; the results forPOMO-SLandSIL-SITare presented in Section 6 and Appendix C of the revised manuscript, respectively.W2: As discussed in Section 7 and Section 8 of our revised manuscript, our findings directly inform three concrete avenues for performance and efficiency improvements:
Probe-Guided Decoding Acceleration of LEHD on Uniform TSP (Section 7, Table 3)
Gap: Optimality gap (%); Steps: Mean decoder forward passes (percentage reduction relative to baseline).
References
[1] Luo et al., "Boosting Neural Combinatorial Optimization for Large-Scale Vehicle Routing Problems," ICLR 2025.