acceptodds
Under review as a conference paper at ICLR 2027

CrossMax: Continual Model Routing with Preferences and Zero-Shot Registration

Abstract

Public model hubs expand continually as new models are published, each requiring discoverability well before evaluation on any real query. Selecting the right model for a query in this setting faces two obstacles. First, users already express preferences in language—a preferred size, language, or publisher—yet routing methods typically ignore it, being trained only to satisfy the functional need, or apply that preference as a filter after scoring. Second, a router must select a model without executing it. Routers rely on past queries, but newly registered models have none. We propose CrossMax, a preference-guided router that supports zero-shot routing. CrossMax relies on two components: a prototype router that trains a small encoder on preference-augmented queries, letting stated preferences shape the ranking directly rather than filter it afterward, and a frozen embedding of the whole model card for models that have never been used. Each component outputs a winner and a margin rule compares the two by their model cards. The margin, set at inference time, chooses how readily a newly registered model is returned. Extending CMRBench, the reference continual routing benchmark, with preferences drawn from model cards, we show that training on the preference text makes the ranking follow the preference, while the same architecture trained without preferences and tested with them does not catch up even when paired with a parsed filter, and that a conservative margin keeps most of the accuracy on models the hub already knows while still returning models that have never been used.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual model routing, preferences, cold-start, model hubs, nearest class mean, compositional representations

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2aHiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies preference-aware routing in a growing model hub, where a model is selected before any candidate is executed. CrossMax combines a preference-trained prototype router for warm models with labeled routing queries and a frozen model-card encoder for cold models without such queries. Each branch selects a winner, and a card-space margin rule chooses between them (lines137–155, 162–215). CMRBench-Pref adds card-derived preference prefixes to the original requests (lines244–252).

The experiments report improved continual-routing accuracy and reduced forgetting on CMRBench and CMRBench-Pref (Table 3, lines335–352). A separate cold-registration evaluation demonstrates a trade-off between retaining warm-model accuracy and correctly selecting cold models, with the conservative margin improving mixed accuracy over warm-only routing (Table 4, lines722–740). The margin is calibrated on an earlier arrival stage and then held fixed (lines998–1001).

Critical strengths

S1. Consistent improvements in continual routing

On original CMRBench, the complete training recipe raises M-Acc from 39.8% to 41.8% and reduces forgetting from 20.1 to 10.5 (Table 3, lines349–352). CMRBench-Pref shows the same direction of improvement (lines335–342). These positive results do not depend entirely on the synthetic preference extension. They support the overall recipe, while differences in replay selection, distillation, and anchoring limit attribution to prototypes alone (lines817–829; Table 7, lines859–862).

S2. A quantified mechanism for admitting cold models

The conservative margin improves mixed accuracy from 33.1% for warm-only routing to 35.7%, while warm accuracy falls from 44.1% to 41.5% and cold accuracy rises from 0% to 18.4% (Table 4, After mask columns, lines722–734). Mixed accuracy covers 2,767 warm and 920 cold queries (lines291–294). The δ sweep and mask ablation expose this trade-off, and the selected margin transfers from an earlier calibration stage (Figure 3, lines392–425; Table 4, lines722–735; lines998–1001). This is a concrete positive result for cold registration in the evaluated setting.

Critical weaknesses

W1. The reliability of the core model-card representation is insufficiently established

Card embeddings determine both the cold winner and the final warm/cold comparison (lines197–215). Their decision reliability is therefore central to the cold-registration contribution, yet the evaluation does not adequately establish the conditions under which this representation is dependable.

W1a. The card ablation does not establish stability under equivalent descriptions. Table 6 varies description length and information content, with different field coverage between the short and richer versions (lines745–755, 786–792, 810–815). It does not test semantically equivalent rewrites of the same model card while holding facts fixed and controlling for length. The results therefore leave sensitivity to wording, ordering, and formatting unresolved.

W1b. Margin calibration does not establish robustness of the underlying representation. Card scores enter both cold-winner selection and the final comparison (lines202–215), so sufficiently large changes in those scores can change either decision. The reported calibration controls the cold-return rate on an earlier stage; it does not evaluate robustness to irrelevant text or unsupported claims (lines998–1001). Clear card-quality assumptions and decision-level reliability evidence are needed to substantiate this central part of cold registration. The paper does not demonstrate instability; the concern is insufficient support for reliability beyond the evaluated card descriptions.

W2. The experiments do not sufficiently isolate preference-aware representation learning

W2a. The main comparison changes several mechanisms at once. The proposed system trains on preference text and uses a lexicon-plus-MLP extractor with a learned penalty; the 41.9% baseline trains on original requests, strips the prefix at evaluation, and uses regex extraction with hard masking (lines259–261, 305–310, 966–993). Thus, the 46.9% versus 41.9% comparison supports the complete system over that baseline, but does not isolate the encoder’s learning of preferences. The proposed result also averages three seeds, while the stripped baseline uses only seed 40 (Table 3, lines325–358). For the same stripped baseline ranking, replacing the regex filter with an oracle filter raises M-Acc from 41.9% to 52.2% (lines989–993). This shows the importance of extraction quality for that baseline; it does not establish that preference training is unnecessary or quantify its independent contribution. Matched extractor/readout controls are needed.

W2b. The drop-prefix diagnostic confounds text removal with penalty removal. The appendix explicitly states that dropping the prefix also removes the penalty (lines978–979). The M-Acc decline from 46.9% to 32.9% therefore measures the combined intervention, not the encoder’s independent use of preference text (lines311–313). It supports the importance of the combined preference signals, while leaving their separate contributions unresolved.

W2c. The GPU-swap result is supporting evidence, but does not resolve the attribution question. The test retains only the GPU field and changes its tier (lines979–980), so it measures GPU-only satisfaction rather than an isolated tier change with other preferences fixed. Reported original-gold M-Acc is 11.8%, and Sat@1 under the new constraint is 37.6% (Table 2, lines271–284; lines313–316). This shows some successful following; avoiding the old gold alone is insufficient, and the result does not separate the encoder’s contribution from the explicit penalty.

Questions for authors

Q1. What supports the decision reliability of model-card embeddings?

Q1a — Clarification. What assumptions about the accuracy, standardization, and trustworthiness of model cards are required by the card-based selection rule? Please clarify the scope of the cold-registration claim under changes in card expression or unsupported capability descriptions (lines197–215, 745–755).

Q1b: Evidence for the current claim. With model facts, requests, and the calibrated δ fixed, how do semantically equivalent card rewrites and ordering changes affect final decision flips and M-Acc? A controlled test would address the reliability gap left by the information-richness ablation (Table 6, lines810–815) and could materially change W1.

Q2. How much improvement remains when extraction and enforcement are controlled?

Q2a: Evidence for the current claim. Using shared extractor outputs ĉ, matched seeds and candidate pools, and the same constraint-scoring rule, how do preference-trained and original-request-trained routers compare? Please control and report whether the prefix is included at inference. Separately, with the encoder and ĉ fixed, what do plain scoring, the Eq. 4 penalty, and hard masking contribute? These controls would clarify the source of the 46.9% versus 41.9% difference (lines305–310, 325–358, 966–993).

Q2b: Evidence for the diagnostic claims. Can you isolate prefix removal from penalty removal and report both M-Acc and Sat@1 (lines311–318, 978–979)? For the GPU-swap test, how should its GPU-only constraint be interpreted relative to the original multi-field request (lines979–980)? A matched tier-change comparison preserving the other fields, where the revised full request remains feasible, would help clarify the encoder’s preference following.

3 comments

We thank the reviewer for the precise questions and ran the controls they describe. Unless stated otherwise, numbers are model accuracy (lower-triangle mean, Eq. 12, seeds 40 to 42).

W1 / Q1. Reliability of the card embeddings

Reviewer: "The reliability of the core model-card representation is insufficiently established [...] What assumptions about the accuracy, standardization, and trustworthiness of model cards are required by the card-based selection rule? Please clarify the scope of the cold-registration claim under changes in card expression or unsupported capability descriptions [...] With model facts, requests, and the calibrated δ\delta fixed, how do semantically equivalent card rewrites and ordering changes affect final decision flips and M-Acc?"

Q1a. The card enters the rule in two ways, under different assumptions.

The structured fields (pipeline tag, publisher, language, size tier, packaging) come from Hub metadata and are taken as accurate. They alone define the feasible set F(c^)F(\hat{c}). A cold model can therefore be chosen only if its metadata satisfies the extracted preference, unless no cold model is feasible.

The free-text description is written by the publisher and is not trusted. It only decides which feasible cold model wins, and the margin comparison.

The card text is rendered by the hub from these fields (Appendix B), so its ordering and format are the hub's choice. The cold-registration claim holds under these assumptions, with δ\delta bounding how often an untried model is returned.

Q1b. We re-embedded the 464 cards of E4, keeping the facts, the requests and the calibrated δ\delta fixed. Seed error on mixed accuracy is ±0.2\pm 0.2 to ±0.3\pm 0.3.

Change to the cardWarm/cold decisions changed (%)Cold winners changed (%)Mixed M-Acc
None, re-embedded (lower bound)0.67.735.7
Description paraphrased, facts kept3.232.735.6
Field order shuffled9.149.034.4
Rendered as prose / key-value / JSON6.5 / 7.1 / 8.442.0 / 42.6 / 46.934.7 / 35.2 / 35.0
Description sentences and field lines reordered8.746.535.2
License and citation text appended4.833.035.5
One unsupported capability sentence added6.034.935.3
Description removed10.557.233.4

The paraphrases were written by Qwen3.8-Flash-Next for 427 cards, with every structured field copied as written. We discarded the 37 rewrites that came back unchanged, too short, or missing one of the original numbers. The kept rewrites are close in length to the originals: the median absolute change is 4.5%, and 85% are within 20%.

The decisions depend on the wording much more than the accuracy does:

  • A paraphrase changes 3.2% of warm/cold decisions and leaves mixed accuracy at the floor.
  • No change that keeps the facts moves mixed accuracy by more than 1.4 points, so most of the changed winners are cold models of similar suitability.
  • An unsupported capability sentence costs 0.4 points.
  • Removing the description costs 2.4 points, because cold accuracy falls from 18.4 to 12.0. Even then the rule stays above warm-only routing (33.1).

Ordering and format, which cause the largest changes, are fixed by the hub.

W2a / Q2a. Isolating preference-aware representation learning

Reviewer: "The main comparison changes several mechanisms at once. [...] Using shared extractor outputs c^\hat{c}, matched seeds and candidate pools, and the same constraint-scoring rule, how do preference-trained and original-request-trained routers compare? Please control and report whether the prefix is included at inference. Separately, with the encoder and c^\hat{c} fixed, what do plain scoring, the Eq. 4 penalty, and hard masking contribute?"

All rows share the same c^\hat{c} (one lexicon and MLP extractor, fitted once and kept fixed), the same seeds and the same 2,125 models.

Encoder trained onText at inferencePreference used asM-AccSat@1
Preference querieswith preferencenothing42.988.1
Preference querieswith preferencepenalty (Eq. 4)45.796.8
Preference querieswith preferencehard mask43.397.8
Preference queries, no penaltywith preferencenothing47.395.5
Original requestswith preferencehard mask38.897.5
Original requests, penalty trained on gold ccwith preferencepenalty40.693.3
Original requestsoriginal requesthard mask47.797.7
Original requests, penalty trained on gold ccoriginal requestpenalty49.094.4

Three conclusions follow:

  1. With the encoder and c^\hat{c} fixed, the penalty is what recovers accuracy (from 42.9 to 45.7). The hard mask gives the highest Sat@1.
  2. On text that contains the preference, an encoder trained on such text (47.3) is ahead of encoders trained on original requests with either readout (38.8 and 40.6). This isolates the contribution of the encoder.
  3. Given the original request, routers trained on original requests are better (47.7 and 49.0). That request, however, cannot be recovered from the written query on 82% of evaluation queries. Removing the preference from the written query reaches at most 41.5, or 44.2 with gold preferences.

W2b / Q2b. Prefix removal, penalty removal and the GPU swap

Reviewer: "The drop-prefix diagnostic confounds text removal with penalty removal. [...] Can you isolate prefix removal from penalty removal and report both M-Acc and Sat@1 [...] For the GPU-swap test, how should its GPU-only constraint be interpreted relative to the original multi-field request [...] A matched tier-change comparison preserving the other fields, where the revised full request remains feasible, would help clarify the encoder's preference following."

Preference-trained router, with c^\hat{c} always extracted from the full query (M-Acc / Sat@1):

Encoder textPenalty onPenalty off
With preference45.7 / 96.842.9 / 88.1
Original request42.6 / 87.832.9 / 62.0

The published 32.9 requires removing both. Either removal alone gives about 43, so the two signals each contribute and partly overlap.

In the GPU swap we change only the tier, keep the other fields, and drop queries whose revised request matches no model:

PenaltyM-Acc on the original goldSat@1 for the new tier
Off39.7±0.739.7 \pm 0.75.6±0.75.6 \pm 0.7
On18.9±2.018.9 \pm 2.048.2±4.248.2 \pm 4.2

Without the penalty the encoder keeps following the original wording. The penalty is what follows the new tier. The published GPU-only test measures a single field, as the reviewer notes, and should be read that way.

Thank you for the additional experiments. They provide a better understanding of model-card reliability and clarify where the gains come from. The new results verify preference training, but the filtering baseline can still perform better when the original request is available. Even if we accept that recovering the original request is difficult, it remains unclear whether the method can obtain the preference information it needs from real user queries. Thus, the practical advantage of preference training over filtering is still not clear.

Reviewer_QrxROpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

CrossMax combines a trained preference-aware prototype router for models with labeled queries with frozen model-card embeddings for newly registered models. A margin rule compares the two winners in the card-embedding space. The paper also introduces CMRBench-Pref to evaluate preference-aware continual routing.

Critical strengths

1.The combination of continual routing, user preferences, and cold-start registration addresses a relevant problem. 2.The prototype router shows encouraging accuracy and forgetting improvements over CARvE at the reported replay fractions. 3.Historical-stage margin calibration, multi-seed results for the main experiments, and explicit cost reporting make the evaluation informative.

Critical weaknesses

1.The benefit of preference-aware training over post-hoc filtering is not fully isolated. The proposed all-warm router uses a lexicon and a trained task head to recover preferences for its score penalty, whereas the parse-then-filter baseline uses regex-based extraction (Appendix G). Consequently, the 46.9% versus 41.9% comparison changes both ranking and extraction, making it difficult to attribute the gain specifically to preference-aware training. The 52.2% oracle-filter result, which the authors acknowledge, further motivates a comparison using identical extracted preferences, although it does not itself establish that practical filtering is superior.

2.The comparative evidence for the cold-start mechanism remains limited. The margin rule is tested on the E3-to-E4 registration transition after calibration on E2-to-E3, and is compared primarily with retrieval and simple score-fusion variants. Its mixed-accuracy improvement from 33.1% to 35.7% is promising, but additional held-out transitions and stronger alignment or gating baselines trained only on historical data would help establish its comparative value under the same no-execution constraint. Although encoding costs are explicitly reported, end-to-end comparisons at matched inference budgets would further clarify the accuracy–cost trade-off of the additional 8B query encoding.

3.Generalization beyond the controlled preference benchmark remains uncertain. Sampling feasible preferences from gold-model cards is a reasonable controlled protocol, and the paper includes vague phrasings and a GPU-swap probe. However, these do not establish performance on independently collected user preferences, particularly when ambiguity or conflicts fall outside the benchmark’s generation rules. Moreover, while Sat@k measures constraint satisfaction, single-gold-model accuracy does not capture the downstream usefulness of alternative feasible selections.

Questions for authors

1.How do preference-conditioned ranking and post-hoc filtering compare when both use identical extracted preferences? 2.Does the cold-start improvement persist across additional held-out registration transitions and stronger alignment or gating baselines under the same no-execution protocol?

1 comment

We thank the reviewer. All numbers are model accuracy (lower-triangle mean, Eq. 12, seeds 40–42).

W1 / Q1. Preference training against post-hoc filtering

Reviewer: "The benefit of preference-aware training over post-hoc filtering is not fully isolated. [...] the 46.9% versus 41.9% comparison changes both ranking and extraction [...] although it does not itself establish that practical filtering is superior. [...] How do preference-conditioned ranking and post-hoc filtering compare when both use identical extracted preferences?"

With one fixed extractor giving the same c^\hat{c} to both routers, filtering is ahead. The router trained on original requests, with a filter, scores 47.7±0.247.7 \pm 0.2. The preference-trained router, with its penalty, scores 45.7±0.445.7 \pm 0.4. With gold preferences the figures are 52.0 and 46.9. The published gap came mostly from the regex extractor, which recovers the full preference on 34.6% of queries, against 87.4% for ours.

The filtered router, however, reads the original CMRBench request, which cannot be recovered from the written query on 82% of evaluation queries. When the preference is instead removed from the written query, filtering reaches at most 41.5, and 44.2 even with gold preferences. Both are below the preference-trained router. With no extractor or penalty, training on queries with preferences gives 47.3±0.047.3 \pm 0.0, against 41.8±0.341.8 \pm 0.3 when trained and tested on the original requests.

A clean preference record is therefore best applied as a filter, and a preference written into the query is best learned. Full numbers are in our reply to Reviewer Yrqq.

W2 / Q2. Evidence for the cold-start mechanism

Reviewer: "The comparative evidence for the cold-start mechanism remains limited. [...] additional held-out transitions and stronger alignment or gating baselines trained only on historical data would help establish its comparative value under the same no-execution constraint. [...] end-to-end comparisons at matched inference budgets would further clarify the accuracy–cost trade-off of the additional 8B query encoding."

δ\delta is calibrated when one group of new models arrives, then applied unchanged to a later group:

δ\delta calibrated on the models ofTested on the models ofWarmColdMixedMixed, warm models only
E2E345.015.235.532.0
E2E442.116.535.733.1
E3 (paper)E441.518.435.733.1

In every case the margin rule improves mixed accuracy over routing only to warm models, by 2.6 to 3.5 points.

We also compared it with baselines fitted only on historical models, without executing any new model. On the models of E4, the margin rule's mixed accuracy (35.7) stays above:

  • a ridge map from card space to prototype space (34.1)
  • a logistic gate on four features (33.9)
  • Platt and isotonic scaling (33.1 and 33.2), which almost never select a new model correctly.

On cost: when the cold winner is always returned, cold accuracy is 29.6, 34.6 and 35.8 with 0.6B, 4B and 8B card encoders. A 4B encode costs about half as much as an 8B one. We have not run the full end-to-end comparison at a matched budget.

W3. Generalization beyond the benchmark

Reviewer: "Generalization beyond the controlled preference benchmark remains uncertain. [...] these do not establish performance on independently collected user preferences, particularly when ambiguity or conflicts fall outside the benchmark's generation rules. Moreover, while Sat@k measures constraint satisfaction, single-gold-model accuracy does not capture the downstream usefulness of alternative feasible selections."

The controlled protocol is what gives every query a known feasible set, and with it Sat@k, the gold-filter bounds and the GPU-swap test. Existing user-preference data for routing records something else: pairwise response preferences (RouterJudge) or domain-action policies (Arch-Router), not stated constraints on model attributes. The phrasing is also not template-like. Two-thirds of the queries are audited rewrites, and on 82% the request cannot be separated from the preference by string matching.

On single-gold accuracy: the top-ranked model is feasible and of the gold model's task on 95.1% of queries, against 45.7% exact matches. In about half of the queries, the router therefore returns a usable alternative that this metric counts as an error. A study with real users is the natural next step.

Regarding Strength 2: the low-replay lead over CARvE comes from the stored query states. A store of one running mean per model and experience (11.7 MB, against 8.7 MB for CARvE) preserves it: 44.5 against 29.7 at 1% replay (see our reply to Reviewer Yrqq).

Reviewer_YrqqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CrossMax which uses a model router for models with labeled queries and embeddings for newly incoming models without those kinds of labels. Experiments on CMRBench-Pref show that explicitly training with high-quality preferences improves the preference-aware routing quality, and the proposed method can route to new models without prior evaluations while preserving most accuracy on known models.

Critical strengths
  1. I think this paper studies an interesting and practical problem of preference constraints and new-model selection.
  2. I think the separation between warm and cold models is well motivated and interesting. The prototype router uses historical query information for established models, while the frozen model-card encoder provides a way to consider new models without first collecting labeled queries or retraining the router.
  3. The experimental results are overall of good quality. The paper also reports warm, cold, and mixed accuracy separately, alongside preference-removal and GPU-swap experiments.
Critical weaknesses

Here is my major concern: The comparison between learning preferences and filtering does not control the preference parser. The proposed method uses lexicon rules together with a trained task MLP, whereas the parse-then-filter baseline uses regex and task-phrase matching. Consequently, I feel the difference between 46.9% and 41.9% cannot be attributed solely to incorporating preferences into ranking. Therefore, I feel some ablation studies on this will make the results more convincing.

Questions for authors
  1. When the same extracted preferences are supplied to both methods, does the preference-trained router still outperform parse-then-filter?
  2. Is there an ablation that removes the fallback to historical states for all old models, rather than only models with replay text? If not, could the small performance drop in Table 5 be explained by the historical states still retained for models without replay text?
2 comments

We thank the reviewer for both questions. We ran both experiments, and each narrows a claim in the paper.

Q1. Same extracted preferences for both methods

Reviewer: "The comparison between learning preferences and filtering does not control the preference parser. [...] When the same extracted preferences are supplied to both methods, does the preference-trained router still outperform parse-then-filter?"

The answer depends on the text the filtered router receives, so we report both cases. We used one extractor (the lexicon rules and the task MLP, fitted once and kept fixed) to give the same c^\hat{c} to both routers. When the filtered router reads the original CMRBench request, as in the paper's baseline, the filter is ahead:

SettingM-Acc
Router trained on original requests, regex filter (as in the paper)41.7±0.141.7 \pm 0.1
Router trained on original requests, filter with shared c^\hat{c}47.7±0.247.7 \pm 0.2
Preference-trained router, penalty with shared c^\hat{c}45.7±0.445.7 \pm 0.4

The published gap came mostly from the extractor. Regex recovers the full preference on 34.6% of queries; the lexicon and MLP extractor recovers it on 87.4%.

The filtered router, however, reads the original CMRBench request, that is, the query with the preference removed perfectly. On 82% of evaluation queries this request cannot be recovered from what the user wrote, because the preference and the request are written together.

We therefore also removed the preference using only the written query, in two ways:

  • deleting the extracted preference phrases;
  • a clause classifier trained on the 699 training queries where the boundary is known.

The best result is 41.5, and neither method exceeds 44.2 even with gold preferences. Both are below the preference-trained router.

Finally, with no extractor and no penalty at all, the encoder trained on queries with preferences reaches 47.3±0.047.3 \pm 0.0, against 41.8±0.341.8 \pm 0.3 when trained and tested on the original requests.

In short:

  • When the preference is a clean, separate record, it is best applied as a filter.
  • When it is written into the query, it is best learned by the encoder.

Q2. Removing the stored states for all old models

Reviewer: "Is there an ablation that removes the fallback to historical states for all old models, rather than only models with replay text? If not, could the small performance drop in Table 5 be explained by the historical states still retained for models without replay text?"

There was no such ablation, and the reviewer's reading is correct: at 1% replay, Table 5 rebuilt 69 models and left 1,133 with their stored states. We withdraw the sentence in §3.3 (lines 375–377).

Replay10%5%1%
CARvE43.537.929.7
No stored states (replay texts only)44.137.728.1
All states stored (submitted, 126.5 MB)46.947.544.7
Running means and counts only, retrained (11.7 MB)47.247.544.5

The lead at low replay comes from what the router retains about past queries, not from the replay exemplars. It does not, however, need every query. Since WW is linear, the prototype can be computed as normalize(Whˉ(m))\mathrm{normalize}(W \bar{h}(m)) from the mean hˉ(m)\bar{h}(m) of the states before projection. A router retrained this way matches the full store at about a tenth of the memory. This is close to CARvE's model store of one learned vector per model (11.7 MB against 8.7 MB), so at 1% replay the lead (44.5 against 29.7) holds at comparable memory.

Thank you very much for your response to my questions and the new ablation results help me understand where the gain comes from better now. After reading other reviewers' opinions on this work, I agree with other reviewers that the gain of training over filtering is not very clear. I will maintain my current score with a relatively low confidence, so please consider my review sparingly.

Comments on the paper

1 comment

Clarification on preference training vs. filtering (to all reviewers and the AC)

We thank the reviewers for engaging with the new results. Because the remaining concern rests on one condition, we want to clarify what that condition means.

The concern is that filtering is better "when the original request is available." In CMRBench-Pref, the original request is the CMRBench query from which the preference query was generated. A deployed router never receives it: the user writes a single query, and the preference is part of it. On 82% of evaluation queries, the original request is not contained in what the user wrote, so it cannot be recovered from the query. The 47.7 for filtering is therefore an oracle condition, like the 52.0 with gold preferences, and not a pipeline that could be deployed.

Except for the oracle row, every pipeline below sees only the query as the user wrote it. All rows share the same preference extractor, the same 2,125 candidate models and seeds 40–42.

PipelineRequiresM-Acc
Encoder trained on preference queriesnothing47.3
Same encoder + penaltypreference extraction45.7
Filter, after deleting the extracted preference phrasespreference extraction + removal from the query41.5
Filter, after removing preference clauses with a trained classifierpreference extraction + removal from the query40.4
Filter, best removal method, gold preferencesperfect extraction + removal from the query44.2
Filter on the stored original request (oracle)not available at deployment47.7

The gold-preference row is the key one. Even with perfect extraction, filtering the written query stays below the trained encoder. A better parser therefore cannot close the gap. The bottleneck is separating the request from the preference, and the trained encoder does not need to do that.

On real user queries (Reviewer 2aHi): we agree that we have no independently collected preferences. We note, though, that this concern applies more strongly to filtering. A filter must both extract the preference and remove it from the query, while the trained encoder needs neither step.

In the final version of the paper, we will scope the claim to preferences stated within the query and state the lack of real-user data as a limitation. We will also recommend filtering explicitly when preferences arrive as a separate structured record, such as UI fields.

We hope this clarifies that the comparison favouring filtering relies on an input unavailable at deployment, while every deployable filtering pipeline we tested falls below the preference-trained encoder. If this addresses the remaining concern, we would be grateful if the reviewers would consider reflecting it in their scores.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.