- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies preference-aware routing in a growing model hub, where a model is selected before any candidate is executed. CrossMax combines a preference-trained prototype router for warm models with labeled routing queries and a frozen model-card encoder for cold models without such queries. Each branch selects a winner, and a card-space margin rule chooses between them (lines137–155, 162–215). CMRBench-Pref adds card-derived preference prefixes to the original requests (lines244–252).
The experiments report improved continual-routing accuracy and reduced forgetting on CMRBench and CMRBench-Pref (Table 3, lines335–352). A separate cold-registration evaluation demonstrates a trade-off between retaining warm-model accuracy and correctly selecting cold models, with the conservative margin improving mixed accuracy over warm-only routing (Table 4, lines722–740). The margin is calibrated on an earlier arrival stage and then held fixed (lines998–1001).
Critical strengths
S1. Consistent improvements in continual routing
On original CMRBench, the complete training recipe raises M-Acc from 39.8% to 41.8% and reduces forgetting from 20.1 to 10.5 (Table 3, lines349–352). CMRBench-Pref shows the same direction of improvement (lines335–342). These positive results do not depend entirely on the synthetic preference extension. They support the overall recipe, while differences in replay selection, distillation, and anchoring limit attribution to prototypes alone (lines817–829; Table 7, lines859–862).
S2. A quantified mechanism for admitting cold models
The conservative margin improves mixed accuracy from 33.1% for warm-only routing to 35.7%, while warm accuracy falls from 44.1% to 41.5% and cold accuracy rises from 0% to 18.4% (Table 4, After mask columns, lines722–734). Mixed accuracy covers 2,767 warm and 920 cold queries (lines291–294). The δ sweep and mask ablation expose this trade-off, and the selected margin transfers from an earlier calibration stage (Figure 3, lines392–425; Table 4, lines722–735; lines998–1001). This is a concrete positive result for cold registration in the evaluated setting.
Critical weaknesses
W1. The reliability of the core model-card representation is insufficiently established
Card embeddings determine both the cold winner and the final warm/cold comparison (lines197–215). Their decision reliability is therefore central to the cold-registration contribution, yet the evaluation does not adequately establish the conditions under which this representation is dependable.
W1a. The card ablation does not establish stability under equivalent descriptions. Table 6 varies description length and information content, with different field coverage between the short and richer versions (lines745–755, 786–792, 810–815). It does not test semantically equivalent rewrites of the same model card while holding facts fixed and controlling for length. The results therefore leave sensitivity to wording, ordering, and formatting unresolved.
W1b. Margin calibration does not establish robustness of the underlying representation. Card scores enter both cold-winner selection and the final comparison (lines202–215), so sufficiently large changes in those scores can change either decision. The reported calibration controls the cold-return rate on an earlier stage; it does not evaluate robustness to irrelevant text or unsupported claims (lines998–1001). Clear card-quality assumptions and decision-level reliability evidence are needed to substantiate this central part of cold registration. The paper does not demonstrate instability; the concern is insufficient support for reliability beyond the evaluated card descriptions.
W2. The experiments do not sufficiently isolate preference-aware representation learning
W2a. The main comparison changes several mechanisms at once. The proposed system trains on preference text and uses a lexicon-plus-MLP extractor with a learned penalty; the 41.9% baseline trains on original requests, strips the prefix at evaluation, and uses regex extraction with hard masking (lines259–261, 305–310, 966–993). Thus, the 46.9% versus 41.9% comparison supports the complete system over that baseline, but does not isolate the encoder’s learning of preferences. The proposed result also averages three seeds, while the stripped baseline uses only seed 40 (Table 3, lines325–358). For the same stripped baseline ranking, replacing the regex filter with an oracle filter raises M-Acc from 41.9% to 52.2% (lines989–993). This shows the importance of extraction quality for that baseline; it does not establish that preference training is unnecessary or quantify its independent contribution. Matched extractor/readout controls are needed.
W2b. The drop-prefix diagnostic confounds text removal with penalty removal. The appendix explicitly states that dropping the prefix also removes the penalty (lines978–979). The M-Acc decline from 46.9% to 32.9% therefore measures the combined intervention, not the encoder’s independent use of preference text (lines311–313). It supports the importance of the combined preference signals, while leaving their separate contributions unresolved.
W2c. The GPU-swap result is supporting evidence, but does not resolve the attribution question. The test retains only the GPU field and changes its tier (lines979–980), so it measures GPU-only satisfaction rather than an isolated tier change with other preferences fixed. Reported original-gold M-Acc is 11.8%, and Sat@1 under the new constraint is 37.6% (Table 2, lines271–284; lines313–316). This shows some successful following; avoiding the old gold alone is insufficient, and the result does not separate the encoder’s contribution from the explicit penalty.
Questions for authors
Q1. What supports the decision reliability of model-card embeddings?
Q1a — Clarification. What assumptions about the accuracy, standardization, and trustworthiness of model cards are required by the card-based selection rule? Please clarify the scope of the cold-registration claim under changes in card expression or unsupported capability descriptions (lines197–215, 745–755).
Q1b: Evidence for the current claim. With model facts, requests, and the calibrated δ fixed, how do semantically equivalent card rewrites and ordering changes affect final decision flips and M-Acc? A controlled test would address the reliability gap left by the information-richness ablation (Table 6, lines810–815) and could materially change W1.
Q2. How much improvement remains when extraction and enforcement are controlled?
Q2a: Evidence for the current claim. Using shared extractor outputs ĉ, matched seeds and candidate pools, and the same constraint-scoring rule, how do preference-trained and original-request-trained routers compare? Please control and report whether the prefix is included at inference. Separately, with the encoder and ĉ fixed, what do plain scoring, the Eq. 4 penalty, and hard masking contribute? These controls would clarify the source of the 46.9% versus 41.9% difference (lines305–310, 325–358, 966–993).
Q2b: Evidence for the diagnostic claims. Can you isolate prefix removal from penalty removal and report both M-Acc and Sat@1 (lines311–318, 978–979)? For the GPU-swap test, how should its GPU-only constraint be interpreted relative to the original multi-field request (lines979–980)? A matched tier-change comparison preserving the other fields, where the revised full request remains feasible, would help clarify the encoder’s preference following.
3 comments
W2a / Q2a. Isolating preference-aware representation learning
Reviewer: "The main comparison changes several mechanisms at once. [...] Using shared extractor outputs , matched seeds and candidate pools, and the same constraint-scoring rule, how do preference-trained and original-request-trained routers compare? Please control and report whether the prefix is included at inference. Separately, with the encoder and fixed, what do plain scoring, the Eq. 4 penalty, and hard masking contribute?"
All rows share the same (one lexicon and MLP extractor, fitted once and kept fixed), the same seeds and the same 2,125 models.
| Encoder trained on | Text at inference | Preference used as | M-Acc | Sat@1 |
|---|---|---|---|---|
| Preference queries | with preference | nothing | 42.9 | 88.1 |
| Preference queries | with preference | penalty (Eq. 4) | 45.7 | 96.8 |
| Preference queries | with preference | hard mask | 43.3 | 97.8 |
| Preference queries, no penalty | with preference | nothing | 47.3 | 95.5 |
| Original requests | with preference | hard mask | 38.8 | 97.5 |
| Original requests, penalty trained on gold | with preference | penalty | 40.6 | 93.3 |
| Original requests | original request | hard mask | 47.7 | 97.7 |
| Original requests, penalty trained on gold | original request | penalty | 49.0 | 94.4 |
Three conclusions follow:
- With the encoder and fixed, the penalty is what recovers accuracy (from 42.9 to 45.7). The hard mask gives the highest Sat@1.
- On text that contains the preference, an encoder trained on such text (47.3) is ahead of encoders trained on original requests with either readout (38.8 and 40.6). This isolates the contribution of the encoder.
- Given the original request, routers trained on original requests are better (47.7 and 49.0). That request, however, cannot be recovered from the written query on 82% of evaluation queries. Removing the preference from the written query reaches at most 41.5, or 44.2 with gold preferences.
W2b / Q2b. Prefix removal, penalty removal and the GPU swap
Reviewer: "The drop-prefix diagnostic confounds text removal with penalty removal. [...] Can you isolate prefix removal from penalty removal and report both M-Acc and Sat@1 [...] For the GPU-swap test, how should its GPU-only constraint be interpreted relative to the original multi-field request [...] A matched tier-change comparison preserving the other fields, where the revised full request remains feasible, would help clarify the encoder's preference following."
Preference-trained router, with always extracted from the full query (M-Acc / Sat@1):
| Encoder text | Penalty on | Penalty off |
|---|---|---|
| With preference | 45.7 / 96.8 | 42.9 / 88.1 |
| Original request | 42.6 / 87.8 | 32.9 / 62.0 |
The published 32.9 requires removing both. Either removal alone gives about 43, so the two signals each contribute and partly overlap.
In the GPU swap we change only the tier, keep the other fields, and drop queries whose revised request matches no model:
| Penalty | M-Acc on the original gold | Sat@1 for the new tier |
|---|---|---|
| Off | ||
| On |
Without the penalty the encoder keeps following the original wording. The penalty is what follows the new tier. The published GPU-only test measures a single field, as the reviewer notes, and should be read that way.
Thank you for the additional experiments. They provide a better understanding of model-card reliability and clarify where the gains come from. The new results verify preference training, but the filtering baseline can still perform better when the original request is available. Even if we accept that recovering the original request is difficult, it remains unclear whether the method can obtain the preference information it needs from real user queries. Thus, the practical advantage of preference training over filtering is still not clear.
We thank the reviewer for the precise questions and ran the controls they describe. Unless stated otherwise, numbers are model accuracy (lower-triangle mean, Eq. 12, seeds 40 to 42).
W1 / Q1. Reliability of the card embeddings
Q1a. The card enters the rule in two ways, under different assumptions.
The structured fields (pipeline tag, publisher, language, size tier, packaging) come from Hub metadata and are taken as accurate. They alone define the feasible set F(c^). A cold model can therefore be chosen only if its metadata satisfies the extracted preference, unless no cold model is feasible.
The free-text description is written by the publisher and is not trusted. It only decides which feasible cold model wins, and the margin comparison.
The card text is rendered by the hub from these fields (Appendix B), so its ordering and format are the hub's choice. The cold-registration claim holds under these assumptions, with δ bounding how often an untried model is returned.
Q1b. We re-embedded the 464 cards of E4, keeping the facts, the requests and the calibrated δ fixed. Seed error on mixed accuracy is ±0.2 to ±0.3.
The paraphrases were written by Qwen3.8-Flash-Next for 427 cards, with every structured field copied as written. We discarded the 37 rewrites that came back unchanged, too short, or missing one of the original numbers. The kept rewrites are close in length to the originals: the median absolute change is 4.5%, and 85% are within 20%.
The decisions depend on the wording much more than the accuracy does:
Ordering and format, which cause the largest changes, are fixed by the hub.