- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces MosaicRec, a benchmark that separates the post-retrieval recommendation pipeline into specification grounding and set construction, enabling separate evaluation of each stage and diagnosis of their interaction. This benchmark can reveal whether an agent correctly grounded these constraints, successfully constructed a compliant set, or recovered from an earlier error. Experiments show that MosaicRec makes individual stages diagnosable while exposing how their interaction shapes final outcomes.
Critical strengths
The paper is well written and easy to follow.
Critical weaknesses
- The separation between the two stages is leaky. In the E2E setting, the LLM constructor can still see the original request and call the tools again.
- The requests follow templates and are clearly specified, while real user requests are often vague.
Questions for authors
- Does the line between F_construct and F_ground depend on how the interface is designed?
- How well would this system work with real user requests?
6 comments
W2: Requests are templated and clearly specified, while real user requests are often vague.
W2: Requests are templated and clearly specified, while real user requests are often vague.
We agree that MosaicRec does not contain requests with several defensible readings. This is a scope decision that follows from what the benchmark measures.
Why every request has a single reading. MosaicRec scores a returned set by checking it exactly against an executable specification, and attributes each failure to grounding or to construction. Both steps require that the request determine one correct specification. If "my favorite genre" could mean most-read, highest-rated, or most recent, a returned set could not be judged compliant or non-compliant without a model judge or overlap with human suggestions, and a failure could not be assigned to the model rather than to the request. Benchmarks built on vague or user-written requests accordingly use those softer measures. We give up open-ended vagueness to obtain exact verification and stage-level attribution.
What the requests do contain. Single-reading does not mean literal. Requests use everyday subjective and relative terms, each tied to one operational reading: "this reader's favorite genre — the one they read most", "lesser-known books — titles with no more than 1508 ratings", "keep it recent: as of 2025, nothing older than 10 years". In many requests the value is never stated and must be derived from the user's history: on Goodreads, 279 of 900 test requests do not name the target genre, and 455 of 900 contain at least one condition whose value is absent from the request.
Requests requiring derived values still produce substantial errors. Splitting the existing end-to-end results by whether the target genre is named or must be derived from the user's history:
| Model | Catalog | CSR, genre named | CSR, genre derived |
|---|---|---|---|
| GPT-5.6-sol | Goodreads | 0.961 | 0.681 |
| Gemma 4 31B | Goodreads | 0.884 | 0.588 |
| Qwen3-32B | Goodreads | 0.580 | 0.333 |
| GPT-5.6-sol | Amazon | 1.000 | 0.621 |
| Gemma 4 31B | Amazon | 0.989 | 0.564 |
This split is descriptive and not a paired comparison: the two groups contain different tasks and may differ in their other constraints. These descriptive results show substantial remaining errors on requests requiring history-derived values, even within our controlled setting. They do not estimate the effect of removing the disambiguating definitions or performance on user-written requests.
Relation to work on vague intent. Recognizing and clarifying ambiguous intent is evaluated by benchmarks such as CRA-Bench and RecBench+. MosaicRec is complementary: it evaluates what happens once the intent is determinate, namely grounding it in distributed evidence and constructing a compliant set. An agent that resolves a vague request through clarification still has to perform these two steps, and our results show substantial remaining errors on them in our setting.
Revision. We will (i) add the table above with per-mode request counts, (ii) add an "ambiguous intent" column to the related-work comparison, marking it as not covered by MosaicRec, and (iii) replace "well-specified requests" in Sec. 6 with a precise statement: every request has a unique reading, but its values need not be stated.
For questions
Q1: Does the line between and depend on how the interface is designed?
Yes. The attribution is defined relative to the specification interface and the stated priority rule (a failure is assigned to the earliest blocking stage), and Sec. 4.1 notes that it is "not a causal decomposition". A different interface, such as free-text notes alongside the specification, could move individual failures between the two categories. We will state this caveat explicitly in Sec. 4.1.
Two properties hold regardless of this dependence:
- Safety is semantic. It compares the sets admitted by the predicted and gold specifications, so two encodings that admit the same sets are classified identically.
- The main construction finding does not use the line. The Gold+LLM comparison fixes the specification across models.
Separately, we checked sensitivity to the grounding criterion. Within the evaluated interface, replacing safety with equivalence changes the conditional construction-failure rate by at most 0.022 (Table 11). This checks sensitivity to the grounding criterion; it does not establish invariance to alternative interfaces.
Q2: How well would this system work with real user requests?
We have not evaluated this, and the paper makes no claim about it. All requests in MosaicRec are generated from specifications, so our results do not measure performance on user-written requests.
What we can say is limited to how the protocol would apply:
- Requests with a determinate reading. The evaluation needs only a request, a candidate pool, and an executable specification. User-written requests annotated with specifications could be scored and diagnosed with the same pipeline, without changes to the checker or the solver.
- Requests with several defensible readings. These cannot be scored exactly as they stand, because no single specification is correct (see our response to W2). They would first need clarification or an annotated set of acceptable readings, which is outside the current scope.
We list evaluation on user-written requests as a limitation in Sec. 6 and will state it more prominently in the revision.
Response to Reviewer qPrD
Thank you for the review. We agree with the factual basis of both weaknesses and explain below why each reflects a deliberate design choice.
W1: The separation is leaky; in E2E the LLM constructor still sees the request and can call tools.
This is correct and intended. E2E is the deployed-agent setting and is not what we use to isolate a stage; Sec. 4.1 states that its CSR "conflates grounding and construction". Isolation comes from three other measurements:
The leak is what the third measurement quantifies. Because the model constructor keeps the request and tools, it can return a valid set from a faulty specification: GPT-5.6-sol does so on 27 of its 112 infeasible specifications, which is why exact execution lowers its CSR. A sealed pipeline could neither exhibit nor measure this recovery. Sec. 4.3 states the information asymmetry explicitly.