acceptodds
Under review as a conference paper at ICLR 2027

MosaicRec: Diagnosing Specification Grounding and Set Construction in Recommendation Agents

Abstract

Modern recommendation agents can draw on user histories, item catalogs, and external tools to satisfy complex user requests. Their task is not merely to rank plausible items, but to construct recommendation sets that jointly satisfy constraints grounded in distributed evidence. End-to-end success alone cannot reveal whether an agent correctly grounded these constraints, successfully constructed a compliant set, or recovered from an earlier error. We introduce MosaicRec, a benchmark that separates the post-retrieval recommendation pipeline into specification grounding and set construction, enabling separate evaluation of each stage and diagnosis of their interaction. MosaicRec contains held-out tasks from Goodreads and an independently constructed Amazon split, each paired with a solver-certified feasible specification. Its constraints include personalized predicates that require joining user histories with catalog metadata. The evaluation crosses predicted and gold specifications with language-model and exact constructors, checks specification safety through counterexample search, and compares model-generated recommendations with exact execution of the same cached specifications. This design reveals stage-specific weaknesses and cross-stage recovery. Under identical common gold specifications, Gemma 4 31B substantially outperforms a similar-sized Qwen3-32B model, revealing a gap in constrained set construction. Separately evaluating intermediate specifications and final recommendations shows that GPT-5.6-sol can produce valid recommendations despite errors in its own specification by retaining access to the original request and tools; replacing this stage with exact execution of its predicted specifications can therefore reduce end-to-end success. These findings demonstrate how MosaicRec makes individual stages diagnosable while exposing how their interaction shapes final outcomes. Code and data are available at an anonymous repository: https://anonymous.4open.science/r/MosaicRec_Anonymous.

Primary area:
datasets and benchmarks
Keywords:
Agents, Recommender, Benchmarking

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_qPrDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MosaicRec, a benchmark that separates the post-retrieval recommendation pipeline into specification grounding and set construction, enabling separate evaluation of each stage and diagnosis of their interaction. This benchmark can reveal whether an agent correctly grounded these constraints, successfully constructed a compliant set, or recovered from an earlier error. Experiments show that MosaicRec makes individual stages diagnosable while exposing how their interaction shapes final outcomes.

Critical strengths

The paper is well written and easy to follow.

Critical weaknesses
  1. The separation between the two stages is leaky. In the E2E setting, the LLM constructor can still see the original request and call the tools again.
  2. The requests follow templates and are clearly specified, while real user requests are often vague.
Questions for authors
  1. Does the line between F_construct and F_ground depend on how the interface is designed?
  2. How well would this system work with real user requests?
6 comments
AuthorsOpenReview ↗

Response to Reviewer qPrD

Thank you for the review. We agree with the factual basis of both weaknesses and explain below why each reflects a deliberate design choice.

W1: The separation is leaky; in E2E the LLM constructor still sees the request and can call tools.

This is correct and intended. E2E is the deployed-agent setting and is not what we use to isolate a stage; Sec. 4.1 states that its CSR "conflates grounding and construction". Isolation comes from three other measurements:

  1. Gold+LLM. The gold specification is injected, so grounding is bypassed and only construction is measured. The Qwen versus Gemma gap (67.7% versus 97.6%) comes from this setting.
  2. Specification safety. A solver examines the written specification by itself, searching for a set it admits that the request forbids. No constructor is involved.
  3. Paired solver replay. The same cached specification is executed by CP-SAT, which sees only the specification and item metadata.

The leak is what the third measurement quantifies. Because the model constructor keeps the request and tools, it can return a valid set from a faulty specification: GPT-5.6-sol does so on 27 of its 112 infeasible specifications, which is why exact execution lowers its CSR. A sealed pipeline could neither exhibit nor measure this recovery. Sec. 4.3 states the information asymmetry explicitly.

AuthorsOpenReview ↗

W2: Requests are templated and clearly specified, while real user requests are often vague.

W2: Requests are templated and clearly specified, while real user requests are often vague.

We agree that MosaicRec does not contain requests with several defensible readings. This is a scope decision that follows from what the benchmark measures.

Why every request has a single reading. MosaicRec scores a returned set by checking it exactly against an executable specification, and attributes each failure to grounding or to construction. Both steps require that the request determine one correct specification. If "my favorite genre" could mean most-read, highest-rated, or most recent, a returned set could not be judged compliant or non-compliant without a model judge or overlap with human suggestions, and a failure could not be assigned to the model rather than to the request. Benchmarks built on vague or user-written requests accordingly use those softer measures. We give up open-ended vagueness to obtain exact verification and stage-level attribution.

What the requests do contain. Single-reading does not mean literal. Requests use everyday subjective and relative terms, each tied to one operational reading: "this reader's favorite genre — the one they read most", "lesser-known books — titles with no more than 1508 ratings", "keep it recent: as of 2025, nothing older than 10 years". In many requests the value is never stated and must be derived from the user's history: on Goodreads, 279 of 900 test requests do not name the target genre, and 455 of 900 contain at least one condition whose value is absent from the request.

Requests requiring derived values still produce substantial errors. Splitting the existing end-to-end results by whether the target genre is named or must be derived from the user's history:

ModelCatalogCSR, genre namedCSR, genre derived
GPT-5.6-solGoodreads0.9610.681
Gemma 4 31BGoodreads0.8840.588
Qwen3-32BGoodreads0.5800.333
GPT-5.6-solAmazon1.0000.621
Gemma 4 31BAmazon0.9890.564

This split is descriptive and not a paired comparison: the two groups contain different tasks and may differ in their other constraints. These descriptive results show substantial remaining errors on requests requiring history-derived values, even within our controlled setting. They do not estimate the effect of removing the disambiguating definitions or performance on user-written requests.

Relation to work on vague intent. Recognizing and clarifying ambiguous intent is evaluated by benchmarks such as CRA-Bench and RecBench+. MosaicRec is complementary: it evaluates what happens once the intent is determinate, namely grounding it in distributed evidence and constructing a compliant set. An agent that resolves a vague request through clarification still has to perform these two steps, and our results show substantial remaining errors on them in our setting.

Revision. We will (i) add the table above with per-mode request counts, (ii) add an "ambiguous intent" column to the related-work comparison, marking it as not covered by MosaicRec, and (iii) replace "well-specified requests" in Sec. 6 with a precise statement: every request has a unique reading, but its values need not be stated.

AuthorsOpenReview ↗

For questions

Q1: Does the line between FconstructF_{\mathrm{construct}} and FgroundF_{\mathrm{ground}} depend on how the interface is designed?

Yes. The attribution is defined relative to the specification interface and the stated priority rule (a failure is assigned to the earliest blocking stage), and Sec. 4.1 notes that it is "not a causal decomposition". A different interface, such as free-text notes alongside the specification, could move individual failures between the two categories. We will state this caveat explicitly in Sec. 4.1.

Two properties hold regardless of this dependence:

  • Safety is semantic. It compares the sets admitted by the predicted and gold specifications, so two encodings that admit the same sets are classified identically.
  • The main construction finding does not use the line. The Gold+LLM comparison fixes the specification across models.

Separately, we checked sensitivity to the grounding criterion. Within the evaluated interface, replacing safety with equivalence changes the conditional construction-failure rate by at most 0.022 (Table 11). This checks sensitivity to the grounding criterion; it does not establish invariance to alternative interfaces.

Q2: How well would this system work with real user requests?

We have not evaluated this, and the paper makes no claim about it. All requests in MosaicRec are generated from specifications, so our results do not measure performance on user-written requests.

What we can say is limited to how the protocol would apply:

  • Requests with a determinate reading. The evaluation needs only a request, a candidate pool, and an executable specification. User-written requests annotated with specifications could be scored and diagnosed with the same pipeline, without changes to the checker or the solver.
  • Requests with several defensible readings. These cannot be scored exactly as they stand, because no single specification is correct (see our response to W2). They would first need clarification or an annotated set of acceptable readings, which is outside the current scope.

We list evaluation on user-written requests as a limitation in Sec. 6 and will state it more prominently in the revision.

Reviewer_3o8XOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces MosaicRec, a benchmark for evaluating whether recommendation agents can construct request-compliant sets from a fixed candidate pool using user requests, interaction histories, and item metadata. It separates specification grounding from set construction, instantiates feasibility-certified tasks on Goodreads and Amazon, and combines specification safety and equivalence checks with gold-specification construction and paired solver replay. The main findings are that construction reliability differs substantially across models and that successful final recommendations can conceal errors in intermediate specifications.

Critical strengths

S1. Semantic evaluation of the intermediate specification is the strongest contribution.

The paper does not equate obtaining one valid answer from a predicted specification with the specification being correct. Instead, counterexample search checks whether that specification admits recommendation sets that violate the request. The distinction between safety and strict equivalence is also appropriate: safety establishes sufficiency for valid construction, whereas equivalence additionally checks whether valid answers have been incorrectly excluded. The appendix reports cross-validation against exhaustive enumeration on small pools. This provides more informative diagnostics than evaluating only the final recommendation or comparing specification fields literally.

S2. The central experiments address distinct questions rather than merely reporting additional end-to-end scores.

S3. The formal quality controls and scope characterization are useful.

Critical weaknesses

W1. The correspondence between natural-language requests and gold specifications needs more direct documentation and validation.

Specification feasibility and recomputability of derived values do not, by themselves, establish that the natural-language request uniquely expresses the specification. For example, Figure 1 uses “outside my familiar genres,” whereas the caption defines the relevant genres as the reader’s most-read main genres apart from the requested tag. That particular interpretation is not uniquely determined by the displayed request. The figure may simply abbreviate the full instructions, but the paper should clarify whether these definitions are available to the model through the actual request, system prompt, or tool documentation, rather than only being encoded in the task generator. The recoverability check in Section 3 is useful, but the manuscript does not make this distinction sufficiently clear.

This also affects interpretation of the rule-based baseline. Rule parser + solver achieves 76.6% and 85.0% across the two catalogs in Table 9, but its parsing rules, knowledge of the request-generation templates, and development-time information are insufficiently described. Strong rule-based performance is not inherently a weakness; however, it makes the language coverage and capability being measured important to characterize. Full request–specification examples, documentation of the rule parser, and an independent semantic audit of a small number of shared failure cases would help. The consensus-misread cases already identified in Appendix F provide a useful starting point. The objective is to distinguish model grounding failures from insufficiently communicated benchmark conventions, not to require a large real-user study.

W2. Recovery is empirically demonstrated, but the role of the intermediate specification remains unclear.

Section 4.3 explicitly states that the model constructor receives the predicted specification, original request, and tool access in a fresh conversation, whereas the solver receives only the specification and metadata. The paired experiment therefore validly compares two execution policies and shows that an erroneous written interface can lose information needed to solve the task. However, the model may either identify and correct errors in the specification or largely disregard it and solve the original request again. The current results do not distinguish these possibilities.

A limited control could remove the predicted specification while keeping the construction-stage request, tools, and budget unchanged. Alternatively, targeted trace analysis could demonstrate how the model uses or corrects an erroneous specification. This would clarify the specification’s actual role in recovery. I do not regard the information asymmetry as invalidating the replay experiment, but the interpretation of cross-stage recovery should remain distinct from the alternative explanation of solving the task again from the original evidence.

Questions for authors

Q1. Are the precise definitions of expressions such as “familiar genres” in Figure 1 fully available to the model? Please provide one complete task including the actual instructions, accessible evidence, and gold specification, and clarify whether the rule parser depends on request-generation templates.

Q2. Is there a control experiment or trace evidence distinguishing correction of the predicted specification from disregarding it and solving the original task again? This distinction affects the interpretation of the recovery mechanism.

Q3. Which exact constructor is used in Table 4, Figure 3, and Table 7? Please clarify where depth-first search and CP-SAT are used, including the feasible-answer selection policy for unsafe specifications.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 3o8X

Thank you for the careful review and for highlighting the value of semantic specification evaluation and the complementary diagnostic experiments. Your questions prompted us to examine the correspondence between the task generator, model-visible instructions, and accessible evidence. This examination identified a target-genre convention and a metadata-visibility issue that affect the interpretation of some grounding results. We describe these issues, distinguish the affected conclusions from those that remain supported, and clarify the recovery evidence and solver implementations. In brief: the safety and equivalence methodology, the gold-specification construction results, and the paired-replay comparison of execution policies are unaffected; the absolute grounding rates on derived-genre tasks and the interpretation of recovery are qualified below. The additional analyses use existing task files and cached rollouts; no additional model runs were performed.

W1 / Q1: Correspondence between requests and gold specifications, model-visible definitions, and the rule parser

We agree with the distinction you draw: feasibility and recomputability from internal metadata do not establish that the model-visible request and evidence adequately specify the intended interpretation.

Task example and model-visible definitions. The following is the full request from Goodreads test task 13:

You help a returning reader find their next books. Use the available tools to inspect the reader's history and the candidate books before deciding. Recommend more books in the single genre this reader reads most often. Rank the reader's history by each book's main genre, set aside the genre you are recommending, and rule out picks whose main genre falls in the top 2 of what remains. Allow no more than 2 books from any single author. Every book must carry the catalog tag 'picture-book'. Include at least 2 lesser-known books — titles with no more than 83 ratings. Return exactly 8 books. Don't recommend books that are already in the reader's history. Only include books published in or after 2005. Skip netgalley titles entirely. Output only the chosen book ids as a set.

The accessible evidence includes 100 pre-cutoff interactions, tools for obtaining history-item and candidate metadata, the 60-item candidate pool, pool summaries and search, and a checker that evaluates a proposed set against the agent's own working specification. Metadata is returned in pages of ten and includes the primary-genre field, with genre-tag and shelf lists truncated to eight entries.

The gold specification requires eight items, at most two per author, publication year ≥ 2005, exclusion of previously read items, inclusion of genre picture-books, exclusion of genre netgalley, the tag picture-book, exclusion of main genres poetry and childrens, and at least two items with no more than 83 ratings.

For the explored-genres condition raised in the review, Figure 1 abbreviates the actual instructions. The request above explicitly specifies counting by main genre, setting aside the requested genre, and excluding the top two remaining genres. The parser's system prompt repeats this rule, and the relevant metadata field is accessible. Numerical meanings of "lesser-known" and "recent" are also stated in the request. We will include the full system prompt, tool schemas, and complete task record alongside this example.

An underspecified target-genre convention. The separate phrase "the single genre this reader reads most often" is not adequately operationalized in the model-visible instructions. The generator counts every catalog genre tag attached to each history item, whereas counting each item's primary genre is also a reasonable interpretation. The request and system prompt do not specify which counting rule is intended. Moreover, on Goodreads the generator uses up to twenty tags per item, while the metadata tool exposes only eight. On Amazon, the generator counts every level of an item's category path except the top level, which the request and system prompt likewise do not state.

This affects the tasks whose target genre must be derived: 279 Goodreads tasks and 264 Amazon tasks. In the cached GPT-5.6-sol predictions, the written genre matches the most frequent primary genre on 258/279 Goodreads tasks and 227/264 Amazon tasks. On Goodreads, applying the intended counting rule to the returned tags fails to reproduce the gold target on 74 of the 279 tasks. The recoverability check in Section 3 uses the full frozen metadata, so it did not detect the model-visible information mismatch.

We therefore agree that disagreement with gold on this subset cannot, by itself, establish a model grounding error. The formal safety and equivalence checks remain valid relative to the encoded gold specifications, but their interpretation as measures of language grounding is affected.

Results avoiding this target-genre inference issue. We separately summarize the existing outputs on tasks that name the target genre explicitly: 621 on Goodreads and 636 on Amazon.

ModelCatalogSafeEquivalentE2E CSR
GPT-5.6-solGoodreads0.8780.8570.961
Gemma 4 31BGoodreads0.8050.7790.884
Qwen3-32BGoodreads0.6880.6470.580
GPT-4oGoodreads0.7180.5830.459
GPT-4o-miniGoodreads0.3530.3350.269
GPT-5.6-solAmazon0.9560.9561.000
Gemma 4 31BAmazon0.9100.9090.989

The Goodreads E2E ordering remains unchanged, and specification correctness and final-answer success remain distinct measurements. These aggregate results do not establish invariance of the failure decomposition or constitute a semantic audit of every remaining predicate.

The common-gold construction comparisons, including Qwen3-32B versus Gemma 4 31B, bypass this target-genre inference step. They remain evidence about construction under supplied executable specifications, as do the gold-specification construction stress tests. We will retain the original full-split results for transparency, report the named- and derived-genre subsets separately, and qualify the affected grounding interpretations.

W2 / Q2: Specification correction versus re-solving the original request

We agree that the current experiment does not distinguish these mechanisms. We have not performed the proposed control that removes the predicted specification while retaining the request, tools, and construction budget.

The cached outputs provide a narrower characterization:

GPT-5.6-solInfeasible specificationsGold-valid final setsAmong these, sets violating only the genre predicate of the written specification
Goodreads1122723
Amazon1203129

Thus, most of these gold-valid outputs satisfy all predicates in the written specification except the genre predicate. This describes how the output departs from the specification; it does not establish whether the other predicates were retained from that specification or independently re-derived from the original request. We will not interpret agreement with those predicates as proof that the constructor used the intermediate specification.

The issue identified under W1 also limits the interpretation of these cases. Of the 112 infeasible Goodreads specifications, 107 occur on derived-genre tasks; all 120 Amazon cases do. This shows concentration in the affected subset, rather than establishing that the convention caused every infeasible prediction. It also means that gold-valid outputs in these cases should not be presented as evidence that the model corrected a misunderstanding of user intent.

We will consequently use "recovery" only in an operational sense: under the original benchmark scoring, the model can return a gold-valid set despite an infeasible written specification, whereas exact execution of that specification cannot produce a feasible set. Paired replay establishes this difference between execution policies. It does not identify specification repair, re-solving, or another process as the mechanism.

We will state these limits explicitly and remove wording that attributes the observed outcomes to correction of the specification.

Q3: Exact constructors and feasible-answer selection policies

The manuscript conflates two constructor implementations. We will correct both their names and the claims associated with them.

  • Table 4, Pred+Solver, and Figure 3. These use a deterministic bounded depth-first constructor on specifications generated by a separate parser rollout. It filters candidates using item-level predicates, searches size-kk subsets with pruning bounds for set-level predicates, and has a budget of 60,000 search nodes. Candidate ordering prioritizes contributions to unmet lower-bound requirements, followed by popularity. It returns the first specification-feasible set found. Failure to find a set is scored as a task failure.

    The node budget matters: a search that reaches the budget without finding a set does not establish infeasibility. This baseline therefore cannot generally be described as construction-error-free or complete. In the cached Pred+Solver runs, the bounded constructor returns no valid set on a small number of safe specifications (for example, 8 of 703 for GPT-5.6-sol on Goodreads). We have not yet confirmed whether the node budget is the cause in each case. We will remove the completeness claims and label this constructor as bounded DFS.

  • Table 4, Safe and Equivalent. These use CP-SAT without a time limit, as described in Appendix A, to establish satisfiability and search for semantic counterexamples.

  • Tables 7–8, paired replay. These use CP-SAT on the same cached specification produced in each E2E rollout. The solver seeks one feasible set, with no objective, one search worker, and no time limit. Returned sets are rechecked by the benchmark evaluator.

Table 4 and Figure 3 also compare systems with separate grounding rollouts. The fixed-specification execution comparison is provided by Tables 7–8, rather than by the difference between the independently grounded cells.

Selection under unsafe specifications. Neither constructor uses the hidden gold specification to select an answer. For a satisfiable unsafe specification, different admitted sets can have different gold validity, so the observed success of one execution depends on its selection policy. For a safe specification, every feasible answer is gold-valid. For bounded DFS, failure to return an answer must additionally be distinguished from a certified infeasibility result.

We will make these distinctions explicit in Section 4.1, Table 3, the relevant captions, and Appendix B. Claims about exact execution of a fixed written specification will be tied to the CP-SAT paired replay, subject to the interpretation limits described above.

Reviewer_CkA2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MosaicRec, a benchmark designed to evaluate recommendation agents handling complex user requests. The proposed framework decomposes the recommendation process into two stages: specification grounding, which translates user requests into explicit, executable constraints using interaction histories and catalog metadata, and set construction, which selects a set of items that jointly satisfies these constraints.

Overall, the paper provides an interesting diagnostic framework for understanding failures in recommendation agents. However, the benchmark tasks are constructed rather than collected from real user requests, raising questions about their representativeness of practical recommendation scenarios. Furthermore, the relationship between constraint satisfaction in the proposed post-retrieval setting and broader recommendation quality remains insufficiently explored.

Critical strengths
  1. A novel perspective on benchmarking recommendation agents
  • While existing recommendation benchmarks primarily focus on final recommendation success, MosaicRec introduces a diagnostic benchmark specifically targeting the post-retrieval stage.
  • By decomposing the recommendation process into specification grounding and set construction, the benchmark provides a structured framework for identifying and analyzing failures at different stages.
  1. Objective and reproducible evaluation framework
  • A key strength of the proposed benchmark is its use of an exact constraint solver (CP-SAT) to verify recommendation constraints and outcomes, rather than relying on LLM-based judges.
  • Each task is certified to have at least one feasible solution, and this design enhances the objectivity and reproducibility of the evaluation process.
Critical weaknesses
  1. Limited representativeness of real-world user requests.
  • Although the benchmark is constructed using real user interaction data, the natural-language requests are generated from predefined gold specifications rather than collected from actual users.
  • Consequently, it remains unclear whether these constructed requests adequately capture the diversity, ambiguity, and complexity of real-world recommendation scenarios. Further validation using authentic user requests would strengthen the practical relevance of the benchmark.
  1. Unclear relationship between post-retrieval performance and recommendation quality.
  • The primary evaluation criterion is whether the recommended items satisfy the specified constraints. However, satisfying these constraints does not necessarily imply that the recommendations align with users' actual preferences or provide high recommendation quality.
  • Notably, the authors report that a half of items in solver-certified valid recommendation sets do not appear in users' observed future interactions. This suggests a potential gap between constraint satisfaction and behavioral relevance.
Questions for authors
  1. The authors report that the most of items in solver-certified valid recommendation sets do not appear in users' observed future interactions. Given this observation, how can the benchmark's constraint satisfaction results be related to recommendation quality measured against actual user behavior? More specifically, is there empirical evidence that models achieving higher CSR also provide better recommendations in terms of user preferences or behavioral relevance?

  2. How would the proposed benchmark and the evaluated models perform when faced with more complex or ambiguous user requests? An analysis of model performance across different difficulty levels would help clarify the robustness of the benchmark and its ability to capture more challenging recommendation scenarios.

  3. GPT-5.6-sol already achieves a perfect observed success rate under gold specifications and across all five construction stress splits, raising questions about the benchmark's ability to distinguish among highly capable models. How would other frontier models from different providers or substantially larger open-weight models perform on MosaicRec?

5 comments

We thank the reviewer for the careful reading and for recognizing the diagnostic framing and the solver-based evaluation as strengths. Several of the concerns below relate to the scope of the benchmark, which the paper states in Sec. 3 ("Agent interface and scope") and Sec. 6 ("Scope and uncertainty"): MosaicRec evaluates whether an agent can turn a fixed candidate pool into a set that complies with an explicit, uniquely interpretable request.

W1: The requests are generated from gold specifications, not collected from users.

This is correct, and we do not claim results on user-written requests. We clarify the scope the benchmark targets and why its requests are constructed.

Scope: explicit, user-stated requests. MosaicRec addresses the setting in which a user states what the recommended set must satisfy. This setting is already deployed. Spotify's Prompted Playlist builds a playlist from a natural-language description that can include or exclude genres, ask for new or familiar music, and refer to the listener's own history (for example, "music from my top artists for the last ten years"). YouTube has tested a prompt-based custom Home feed, and Instagram's "Your Algorithm" lets users type the topics they want to see more or less of. In each case the output must comply with what the user asked for. MosaicRec evaluates this compliance. It does not address requests whose intent must first be elicited or clarified, and it does not cover the full diversity of how users phrase requests.

Why the requests are constructed. MosaicRec checks a returned set exactly against an executable specification and attributes each failure to grounding or to construction. Both steps require that a request determine one correct specification. A request with several defensible readings could only be scored with a model judge or by overlap with human suggestions, and its failures could not be attributed to a stage. Generating each request from its specification guarantees this correspondence; collecting requests from users would require annotating each with a specification, which we have not done.

This trade-off is shared by verifiable benchmarks. Benchmarks that score recommendations with exact checks also construct their requests. RecBench+ generates its queries with a language model from attributes extracted from item knowledge graphs. τ-Rec builds its tasks from constraints that can be re-validated against a catalog and uses a language-model user simulator. Datasets that do contain user-written requests, such as Reddit-Movie, score a recommendation by whether it matches items mentioned in community replies, which does not determine whether a returned set complies with the request. We are not aware of a recommendation benchmark that combines user-written requests with exact verification of set-level constraints. MosaicRec does not close this gap; it takes the verifiable side of the trade-off and adds set-level and cross-source constraints with stage-level diagnosis.

Explicit does not mean literal. On Goodreads, 279 of 900 test requests do not name the target genre ("this reader's favorite genre — the one they read most"), and 455 of 900 contain at least one condition whose value is absent from the request and must be derived from the user's history. Splitting the existing end-to-end results accordingly:

ModelCatalogCSR, genre namedCSR, genre derived
GPT-5.6-solGoodreads0.9610.681
Gemma 4 31BGoodreads0.8840.588
Qwen3-32BGoodreads0.5800.333
GPT-5.6-solAmazon1.0000.621
Gemma 4 31BAmazon0.9890.564

This split is descriptive and not a paired comparison: the two groups contain different tasks and may differ in their other constraints. It shows substantial remaining errors on requests requiring history-derived values within our controlled setting. It does not estimate performance on user-written requests.

Revision. We will state this scope in the introduction and in Sec. 3, add the comparison of request sources to the related-work discussion, and list validation on user-written requests as a limitation in Sec. 6.

W2 / Q1: Satisfying constraints does not imply that recommendations match user preferences. Is there evidence that models with higher CSR also give more behaviorally relevant recommendations?

We agree with the premise: satisfying the stated constraints does not imply that a set matches the user's preferences, and CSR is not intended as a measure of preference alignment.

What is in scope. MosaicRec starts after candidate retrieval and evaluates one question: given a fixed pool and an explicit request, does the returned set comply with the request? Sec. 3 lists relevance learning as out of scope, and the paper reports overlap with future interactions "for context only" (Table 5) and as "context rather than the target" (Appendix B). The benchmark therefore does not rank compliant sets by how well they match the user's taste, and it makes no claim that a higher CSR means better preference prediction.

Why compliance matters within this scope. For an explicit request, compliance is a necessary condition for a good recommendation, not a sufficient one. A set that violates what the user asked for ("nothing I have already read", "no more than one book per author", "at least two lesser-known titles") fails the request however relevant its items are. Several of these conditions are properties of the whole set: quotas, coverage, and long-tail requirements cannot be determined by any item-level relevance score. This is the part of recommendation quality that MosaicRec measures and that overlap with logged behavior does not.

Many requests explicitly ask to depart from past behavior. On Goodreads, 445 of 900 test requests require authors the reader has not read before, 625 require a minimum number of lesser-known titles, and 92 exclude the genres the reader reads most; 804 of 900 contain at least one of these conditions (815 of 900 on Amazon). For such requests, similarity to the user's history is not the criterion the user set, and for some of them it points the wrong way: a recommender that favors familiar authors or popular items would be relevant by conventional measures and would violate the request. Whether the returned set honors these conditions is what CSR measures.

What the paper reports as context. Table 5 gives Recall@10 against each user's observed future items next to CSR:

End-to-end agentCSRRecall@10 vs. observed future items
GPT-4o-mini0.2580.287
GPT-4o0.4290.352
Qwen3-32B0.5030.375
Gemma 4 31B0.7920.461
GPT-5.6-sol0.8740.534

The five models are ordered identically by the two measures, so on this benchmark they do not conflict. We do not present this as validation of CSR. Each gold specification is derived from a feasible set of the user's later highly rated items, so compliant sets are drawn from a region of the pool enriched in those items, and the agreement is partly structural. The same construction explains the figure the reviewer cites: 42.6% of items in a solver-certified valid set are absent from the observed future interactions, and these items are unobserved, not known to be disliked.

What we have not done. We have not measured user satisfaction or online behavior for compliant versus non-compliant sets. Relating compliance to preference alignment would require such a study and is outside the scope of this paper.

Revision. We will state in the abstract and in Sec. 3 that CSR measures compliance with an explicit request and is not a measure of preference alignment, move the future-overlap results and this discussion into the main text, report the counts of requests that ask to depart from past behavior, and list the missing link to user-level outcomes as a limitation in Sec. 6.

Thank you for the authors’ response. The additional analysis and experiments provided by the authors have addressed my main concerns, and I am willing to raise my score to 3.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the revision

We thank the AC and the reviewers for their feedback. We have uploaded a revised PDF. All changes are marked in purple. No new model runs were performed: every added number comes from the existing task files and cached rollouts.

1. Scope stated explicitly (Reviewers CkA2, qPrD)

  • The abstract, introduction, and Sec. 3 now state that MosaicRec targets explicit requests that are generated from their specifications and have a single executable reading. Ambiguous requests, user-written requests, and preference prediction are listed as out of scope.
  • Sec. 3 states that CSR measures compliance with an explicit request and is not a measure of preference alignment.
  • Appendix C adds a paragraph on compliance versus behavioral overlap: compliance is necessary but not sufficient; CSR and Recall@10 order the five models identically, but this agreement is partly structural; and many requests ask to depart from past behavior (counts given).
  • Sec. 6 replaces "well-specified requests" with this scope and adds two limitations: user-written requests are not evaluated, and CSR is not related to user satisfaction or online behavior.
  • Related work notes that, like RecBench+ and τ-Rec, MosaicRec constructs its requests.

2. A target-genre convention that was not communicated to the model (Reviewer 3o8X)

Reviewer 3o8X's questions led us to compare the task generator with the instructions and evidence visible to the model. We found that the "most-read genre" is resolved by the generator with a counting rule that neither the request nor the system prompt states, and that on Goodreads the metadata tool returns eight tags per item while the generator counts up to twenty. This affects the 279 Goodreads and 264 Amazon test tasks whose target genre must be derived, and it qualifies the grounding rates reported on the full split.

  • New Appendix H documents a complete task (request, accessible evidence, gold specification), states which definitions are available to the model, describes the convention and its measured effect, and reports all end-to-end results separately for named-genre and derived-genre tasks (new Table 15, five models, both catalogs).
  • Sec. 3 notes that the recoverability check uses the full frozen metadata and not the model-visible evidence.
  • Sec. 5.2 reports the named-genre results next to the full-split rates, and Sec. 6 lists the convention as a limitation.
  • Appendix H documents the rule parser as template-aware, with access to untruncated metadata.

Unaffected by this issue: the safety and equivalence methodology, the gold-specification construction results (including Qwen3-32B versus Gemma 4 31B), the construction stress splits, and the paired-replay comparison of execution policies. Affected: the absolute grounding rates on derived-genre tasks and the interpretation of recovery.

3. "Recovery" is now used only operationally (Reviewers 3o8X, qPrD)

  • The abstract, introduction, and Sec. 5.3 no longer say that a model corrects errors in its own specification. They state the observed fact: a model constructor that retains the request and tools can return a gold-valid set where exact execution of its written specification cannot.
  • Sec. 4.3 and Sec. 5.3 state that the paired replay compares two execution policies and does not identify whether the constructor repairs the specification or solves the request again. The control proposed by Reviewer 3o8X has not been run.
  • Appendix H adds trace evidence for tasks with infeasible specifications (new Table 16) and states its limits, including that 107 of GPT-5.6-sol's 112 infeasible Goodreads specifications arise on derived-genre tasks.
  • The title of Sec. 5.3 is changed accordingly.

4. Constructor naming corrected (Reviewer 3o8X)

  • The Pred+Solver cell (Table 4, Figure 3) uses a deterministic depth-first constructor with a node budget on a separate parser rollout, not CP-SAT. Sec. 4.1, Table 3, and the captions of Table 4 and Figure 3 are corrected, and the claim that this cell is free of construction error is removed.
  • Appendix B describes this constructor (ordering, budget, failure handling), states that it is not complete under the budget, and reports the cases in which it returned no valid set on a safe specification.
  • CP-SAT is used for the safety and equivalence checks and for the paired replay (Tables 7–8). Sec. 5.3 now points to the paired replay for the fixed-specification comparison.

5. Changes to the abstract

Three edits, all following from the points above: one added sentence stating the scope; "cross-stage recovery" changed to "cross-stage interaction"; and the sentence on GPT-5.6-sol reworded to the operational description in point 3. The contributions and reported numbers are unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.