acceptodds
Under review as a conference paper at ICLR 2027

FOODBOUNDARYBENCH: DO CONVERSATIONAL LLMS HOLD HEALTH BOUNDARIES ACROSS MULTI- TURN RECOMMENDATIONS?

Abstract

Users increasingly seek preference-aligned and health-aware food guidance from large language model (LLM) assistants in multi-turn dialogues. However, existing health-aware food recommendation benchmarks typically score a single final recommendation, obscuring whether assistants select the appropriate recommendation policy and preserve it when users continue to press for their preferred food. We introduce FoodBoundaryBench, a trajectory-level benchmark with 1,026 official-guidance-grounded disease–food dialogue scenarios spanning five health-conditioned recommendation policies: rejection, safer-variant redirection, clarification, guarded conditional advice, and individualized deferral. To test whether these policies remain stable under realistic preference pressure, we propose Decoupled Strategy and Live Realization (DSLR), an auditable adaptive-dialogue protocol that fixes scenario-level pressure objectives while allowing user turns to adapt to the target assistant’s replies. Across eight contemporary LLMs, no model exceeds 67.3% final-answer policy pass or 51.1% all-turns policy pass, and final-answer scoring exceeds all-turns scoring by 13.5–20.2%. These results show that both selecting the correct recommendation policy and preserving it under multi-turn preference pressure remain challenging.

Primary area:
datasets and benchmarks
Keywords:
Conversation system, multi-step evaluation, Large language models

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 2 reviews, above 9% of the others.
Reviewer_7AuUOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces FoodBoundaryBench, a benchmark for evaluating whether conversational LLMs select and maintain appropriate health-conditioned food recommendation policies across multiple turns. It contains 1,026 scenarios grounded in official guidance and organized into five policy categories. The proposed DSLR protocol fixes scenario-level pressure objectives while allowing simulated user follow-ups to adapt to model responses. Evaluation combines structured response extraction with deterministic policy adjudication and reports both final-answer and all-turns compliance. Experiments across eight models and two prompting conditions show that a correct final answer can conceal earlier policy violations. Human audits and sensitivity analyses provide additional validation of this finding.

Critical strengths
  1. A useful domain-specific evaluation resource. The benchmark grounds its scenarios in official guidance and distinguishes five appropriate recommendation policies, including clarification, safer alternatives, and conditional advice. This provides a more detailed assessment of food recommendation behavior than a binary refusal-based evaluation and gives the dataset practical value for studying health-aware conversational assistance.

  2. Meaningful validation of the empirical findings. The paper includes human audits, evaluator checks, and sensitivity analyses. In particular, human annotators reproduce a final-versus-all-turns compliance gap similar to the automatic estimate. This strengthens confidence that the reported earlier-turn violations reflect identifiable model behavior.

Critical weaknesses
  1. The incremental contribution over existing multi-turn safety evaluation is not sufficiently established. Prior benchmarks already assess whether models maintain appropriate behavior under conversational pressure. SYCON Bench evaluates expected behavior at each turn, while SafeDialBench uses the lowest turn-level safety score to assess a dialogue. The official-guidance grounding and five food-specific policy categories are useful refinements, but the paper does not clearly establish what additional diagnostic value they provide beyond existing approaches. This limits the significance of the contribution, particularly because the benchmark and evaluation protocol constitute the paper’s main contributions.

  2. The empirical coverage and intervention analysis provide limited generalizable insight. The evaluation concentrates on eight flagship model services, without representative small open-weight models or comparisons across model scales. The selection of model versions could also be better justified given the availability of subsequent releases. Although the paper includes explicit-boundary prompting, retrieval, and self-checking baselines, the supplementary retrieval and self-checking experiments cover only 200 scenarios and one or two target models. The aggregate improvements do not adequately explain which policy categories benefit, whether the gains reflect better initial policy selection or subsequent policy preservation, or why intervention effectiveness differs across models. These limitations reduce the benchmark’s informativeness about the sources of failure and promising directions for improvement.

Questions for authors

Please address the clarification points raised in the weaknesses above, particularly the benchmark’s incremental diagnostic value over prior work, the rationale for model selection, and what the existing intervention results reveal about initial policy selection versus subsequent policy preservation.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 7AuU (1/4)

We thank the reviewer for recognizing the value of the source-grounded recommendation policies and the validation of earlier-turn failures. We address the two concerns below: the additional diagnostic information provided by the benchmark, and the breadth and interpretation of the empirical results.

1. Incremental diagnostic value beyond existing multi-turn safety evaluation

FoodBoundaryBench was designed to evaluate safe food recommendation in multi-turn interactions. Success in this setting requires both respecting source-grounded food–health safety constraints and completing the recommendation actions required by the scenario. Both requirements are already included in our original task definition and Policy Pass criterion (Section 2).

SYCON Bench examines sycophantic conformity and stance changes under user pressure, while SafeDialBench evaluates unsafe-information identification, handling, and safety consistency under jailbreak attacks. FoodBoundaryBench focuses on the continued fulfillment of scenario-specific, source-grounded recommendation policies. These policies specify actions such as providing an acceptable safer variant, clarifying missing food details before giving permission, or preserving the conditions governing a recommendation. Appendix C already illustrates how a response can preserve a safety stance while remaining incomplete under the required recommendation policy.

To make this original evaluation scope and its diagnostic value clearer, we added a separate analysis of safety and recommendation-action completeness to the existing Policy Fail results. We distinguish unsafe and incomplete responses, unsafe but complete responses, and safe but incomplete responses, and report their proportions across the five policy categories. This additional analysis makes explicit which components of the original task requirements are unmet and how the failure patterns differ across policies.

Policy bucketOriginally failed repliesUnsafe and incompleteUnsafe but completeSafe but incomplete
Reject421386 (91.69%)3 (0.71%)32 (7.60%)
Safer Variant1,219878 (72.03%)227 (18.62%)114 (9.35%)
Clarify-first304205 (67.43%)71 (23.36%)28 (9.21%)
Conditional16388 (53.99%)51 (31.29%)24 (14.72%)
Individualized628400 (63.69%)103 (16.40%)125 (19.90%)
All five policies2,7351,957 (71.55%)455 (16.64%)323 (11.81%)
Safer Variant: eight-model extension9,1496,760 (73.89%)1,694 (18.52%)695 (7.60%)

The results reveal distinct failure patterns. Reject failures predominantly involve both unsafe advice and incomplete recommendation actions (91.69%). For Safer Variant, the eight-model analysis assigns 73.89% of failures to unsafe and incomplete responses and 18.52% to unsafe responses despite complete recommendation actions; 7.60% are safe but incomplete. Thus, otherwise-safe responses with incomplete actions do not account for most of the observed Safer Variant failures.

The separation is particularly informative for the more context-dependent policies. Unsafe but complete responses account for 23.36% of Clarify-first failures and 31.29% of Conditional failures. These cases illustrate why carrying out the expected recommendation action does not by itself establish the safety of the resulting advice. Conversely, Individualized has the largest safe but incomplete share (19.90%), identifying failures in which avoiding unsafe advice and fully satisfying the required recommendation actions come apart.

This decomposition makes the diagnostic role of the policy categories explicit: it identifies both the safety status of a failed response and the recommendation requirement involved. It distinguishes unsafe advice that also omits required actions, unsafe advice despite complete actions, and otherwise-safe responses with incomplete actions. These distinctions provide more specific information about what needs improvement than an aggregate Policy Pass rate alone.

AuthorsOpenReview ↗

Response to Reviewer 7AuU (2/4)

2. Expanded empirical coverage and intervention analysis

We have expanded the evaluation and organized the additional analyses around the distinct issues raised in the review. Throughout, Initial Pass refers to the first eligible response, Final Pass to the final eligible response, and All-turns Pass to the complete scored trajectory. Percentage-point changes compare conditions on the same scenarios.

2.1. Open-weight models on a shared 200-scenario subset

We added Qwen3.5-9B, Qwen3.5-27B, Qwen3.8-27B, and Gemma4-31B-it, extending the evaluation to a smaller 9B model, a same-generation 9B/27B comparison, a subsequent-generation 27B model, and another open-weight model family.

Given the rebuttal time constraints, we prioritized a comparison on the same fixed 200-scenario subset used in the paper's supplementary intervention experiments. This subset covers all five policy categories. Reusing it preserves scenario-level comparability across models and aligns the Baseline results with the additional Explicit Boundary evaluation below.

All four models yielded 200 valid Baseline results:

ModelInitial PassFinal PassAll-turns Pass
Qwen3.5-9B63.0%54.5%32.0%
Qwen3.5-27B67.5%53.5%34.0%
Qwen3.8-27B67.0%56.0%35.5%
Gemma4-31B-it64.5%53.0%28.5%

These results extend the model coverage beyond the original flagship services and show that the gap between initial, final, and whole-dialogue performance also occurs in the added open-weight models.

2.2. Comparisons across model sizes and versions

Given the limited time available for the rebuttal, we focus this additional analysis on two complementary comparisons using the same fixed 200-scenario subset as in the paper's supplementary experiments. Reusing this subset preserves scenario-level comparability across model sizes and versions. All comparisons use the Baseline condition and a shared auxiliary-model configuration. Qwen3.5-9B versus Qwen3.5-27B provides a same-generation size comparison, while Qwen3.5-27B versus Qwen3.8-27B provides a version comparison at the same nominal parameter scale.

ComparisonInitial Pass differenceFinal Pass differenceAll-turns Pass difference
Qwen3.5-9B → Qwen3.5-27B+4.5−1.0+2.0
Qwen3.5-27B → Qwen3.8-27B−0.5+2.5+1.5

In the size comparison, the 27B model has a higher observed Initial Pass, while the All-turns difference is smaller. In the version comparison, Initial Pass is similar (67.5% versus 67.0%), while Qwen3.8-27B has slightly higher observed Final Pass (56.0% versus 53.5%) and All-turns Pass (35.5% versus 34.0%). These results extend the evaluation across both model sizes and versions while distinguishing initial, final, and whole-dialogue performance.

2.3. Model versions and evaluation on a subsequent release

To address the availability of subsequent releases directly, we added GPT-6.1 Sol, evaluating Baseline and Explicit Boundary on the same fixed 200 scenarios. All 200 results in each condition were valid.

MetricBaselineExplicit BoundaryChange
Initial Pass70.0%71.5%+1.5
Final Pass50.0%64.0%+14.0
All-turns Pass35.0%53.5%+18.5

The newly evaluated release continues to exhibit a substantial difference between final-answer success and success throughout the interaction, while benefiting from Explicit Boundary.

The original eight-model cohort was selected to provide broad coverage across model families during the main evaluation of 21–24 May 2026.

AuthorsOpenReview ↗

Response to Reviewer 7AuU (3/4)

2.4. Expanded retrieval and self-check evaluation, with an explicit sampling rationale

Retrieval. We expanded the analysis of archived GPT-5.5 BM25 trajectories from the original 200-scenario appendix subset to the full benchmark. All-turns Pass increases from 47.0% to 53.6%, a gain of 6.7 points. Among the usable scenarios outside the original 200 subset, the gain is 7.2 points. This directly checks whether the original-subset finding extends to the remainder of the benchmark.

Self-check. We completed a new GPT-5.5 study on 400 distinct scenarios, comprising the original 200 and a pre-fixed, non-overlapping extension of 200.

Self-check cohortBaseline → Self-check All-turns PassGain
Original subset40.0% → 48.5%+8.5
Non-overlapping extension37.0% → 49.5%+12.5
Combined designed sample38.5% → 49.0%+10.5

The extension was sampled with a fixed random seed from the 826 scenarios outside the original subset, with 40 scenarios per policy category. Given the rebuttal time budget, we prioritized completing this matched extension to test the findings on additional scenarios while preserving coverage of every policy category. Both conditions use the same target and auxiliary-model configuration, and the simulator sees the revised response in Self-check.

2.5. Initial Pass versus subsequent preservation

We now examine these stages for Explicit Boundary, retrieval, and self-check.

Across the original eight models and 1,026 scenarios, Explicit Boundary changes aggregate Initial Pass from 65.2% to 64.0%, while All-turns Pass increases from 35.3% to 45.0%. Among the 4,178 model–scenario pairs that pass initially under both conditions, subsequent failure decreases from 41.6% to 27.7%. This supplies evidence about preservation after an initially satisfactory response.

The added open-weight experiments show the same distinction. Initial Pass changes by −3.5, +1.5, +1.5, and −3.0 percentage points, respectively. To examine subsequent policy preservation independently of differences in initial pass status, we focus on scenarios where the model achieves Initial Pass under both Baseline and Explicit Boundary. Among these shared initially passing scenarios, subsequent failure rates decrease from 41.1% to 30.0%, 43.2% to 27.9%, 43.4% to 23.6%, and 45.7% to 33.0%, respectively. For GPT-6.1 Sol, the same analysis covers 126 scenarios, with subsequent failure decreasing from 46.8% to 23.8%.

Retrieval has a different observed profile. For GPT-5.5 BM25, Initial Pass increases from 65.0% to 71.5%; among the 605 scenarios passing initially under both conditions, subsequent failure decreases from 24.8% to 19.8%. In the new 400-scenario Self-check study, Initial Pass changes by only +1.5 points, while Final Pass and All-turns Pass improve by 11.5 and 10.5 points.

AuthorsOpenReview ↗

Response to Reviewer 7AuU (4/4)

2.6. Which policy categories benefit, and where trade-offs remain

For Explicit Boundary across the original eight models, All-turns Pass gains are 19.4 points for Reject, 15.0 for Safer Variant, 18.6 for Clarify, and 13.5 for Conditional, while Individualized decreases by 7.8 points. Initial Pass in Individualized decreases by 24.2 points, helping explain why the overall Initial Pass does not improve despite gains in several other categories.

The new open-weight results provide a concrete preservation example. For Qwen3.5-27B in Clarify, Initial Pass remains 89.6%, but All-turns Pass increases from 47.9% to 77.1%. Among the 39 scenarios passing initially under both conditions, subsequent failure decreases from 46.2% to 15.4%.

They also reveal trade-offs: Individualized Initial Pass decreases from 77.6% to 56.9% for Qwen3.5-9B and from 91.4% to 55.2% for Gemma4-31B-it. All-turns Pass changes are mixed across the four added open-weight models, showing no consistent improvement in this category. This effect is not universal across targets: GPT-6.1 Sol's Individualized All-turns Pass increases from 44.8% to 74.1%.

For retrieval, Clarify has the largest observed full-benchmark BM25 All-turns gain, 12.6 points; the Conditional is −1.8 points. In the independent 200-scenario Self-check extension, Reject and Safer Variant improve by 27.5 and 20.0 points, while the Conditional is −10.0 points.

These results identify which recommendation requirements improve, where initial execution becomes harder, and where the direction of the effect depends on the model or intervention.

2.7. Why observed intervention gains differ across models

We decompose net All-turns Pass gains into Fail → Pass improvements and Pass → Fail regressions. Both percentages use all paired scenarios for the corresponding model as the denominator.

This analysis covers all eight original models. For example, under Explicit Boundary, GPT-5.5 has 13.3% improvements and 9.2% regressions; Gemini 3.1 Pro has 19.7% improvements and 16.0% regressions; and Kimi K2.6 has 25.8% improvements and 10.4% regressions, each over 1,026 scenarios. Thus, a small net gain can reflect relatively few improvements or a larger offset from regressions.

The same analysis extends to all five newly added Explicit Boundary targets:

ModelPaired scenariosFail → PassPass → FailNet All-turns Pass gain
Qwen3.5-9B20019.0%11.0%+8.0
Qwen3.5-27B20023.5%9.5%+14.0
Qwen3.8-27B20022.0%10.0%+12.0
Gemma4-31B-it20022.0%9.0%+13.0
GPT-6.1 Sol20027.0%8.5%+18.5

Improvements and regressions coexist in every added target. For example, Qwen3.5-27B's larger observed gain relative to Qwen3.5-9B combines a larger improvement share with a smaller regression share.

The self-check results provide another example: GPT-5.5 has 17.5% improvements and 6.5% regressions, whereas DeepSeek-V4-Pro has 16.0% and 14.5%, respectively, each over the same 200 scenarios. Their different net gains, 11.0 versus 1.5 points, therefore accompany similar improvement shares but substantially different regression shares.

This decomposition explains the outcome changes behind the aggregate effects under the evaluated configurations. Together with the stage and category analyses, it supports evaluating interventions for both their ability to satisfy previously unmet recommendation requirements and their ability to preserve requirements already satisfied.

Reviewer_CfGvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces FoodBoundaryBench, a benchmark for evaluating whether LLMs can provide appropriate food recommendations and maintain them across multi-turn conversations. It includes 1,026 scenarios covering five recommendation policies. The authors propose DSLR to generate conversations in which simulated users repeatedly challenge the model's recommendations. They evaluate eight LLMs using both final-answer and all-turns policy compliance. The benchmark is validated through human annotation, simulator-quality checks, and ablation studies.

Critical strengths

1. A meaningful and practical problem. I find the research question important and somewhat overlooked. Checking only the final response can miss meaningful failures. I think evaluating the entire conversation is a valuable direction.

2. Large benchmark and validation. The authors develop 1,026 scenarios with different recommendation policies and evaluate eight LLMs. I also appreciate the human auditing and additional robustness experiments.

Critical weaknesses

1. Not fully convinced by the scoring criteria and the interpretation of the results.

The benchmark requires models to satisfy specific recommendation policies, which sometimes go beyond answering the user's question safely. For example, a model may correctly advise against eating a particular food but still fail the safer-variant policy because it does not offer an alternative. I understand the motivation, but I think safe but incomplete recommendations should be clearly distinguished from true unsafe advice. This is important because the paper shows very low pass rates, especially for the safer-variant category. Although the authors provide additional error diagnostics, I still find the interpretation of the overall scores somewhat difficult.

2. The novelty and added value of the evaluation framework are not sufficiently clear to me.

Multi-turn sycophancy and failures under repeated user pressure have already been studied. The food recommendation setting is useful, but I am less convinced that the relatively complex DSLR framework provides very new insights.

The experiments also lack a direct comparison with ordinary multi-turn conversations without targeted pressure.

3. The presentation can be improved. I found the paper unnecessarily difficult to read, with many terms, symbols. Figure 2 is crowded, and the layout of Figure 1 also makes the paper harder to follow. I think the paper could explain its task and evaluation more simply.

Questions for authors
  1. How much of the low policy pass rate comes from unsafe recommendations, versus safe responses that fail to satisfy all the required recommendation actions? To better understand safer-variant category.

  2. DSLR is a complicated dialogue generation framework. Could a simpler LLM-based user simulator achieve similar results? I found it difficult to understand what the additional components of DSLR actually contribute.

4 comments
AuthorsOpenReview ↗

Response to Reviewer CfGv (1/2)

We thank the reviewer for the constructive feedback and for raising important questions about the evaluation criteria and DSLR's added value.

1. How much of the low policy pass rate comes from unsafe recommendations, versus safe responses that fail to satisfy all the required recommendation actions?

We audited all 1,026 scenarios across all five policies for GPT-5.5 and DeepSeek-V4-Pro under the baseline prompt. The 2,052 dialogues contained 7,088 eligible assistant replies, of which 2,735 failed the original strict score. For each reply, we assessed whether the advice is unsafe and whether the required recommendation action was complete.

Policy BucketAll failed repliesUnsafe and incompleteUnsafe but completeSafe but incomplete
Reject421386 (91.73%)3 (0.78%)32 (7.49%)
Safer Variant1,219878 (72.00%)227 (18.67%)114 (9.33%)
Clarify-first304205 (67.48%)71 (23.30%)28 (9.22%)
Conditional16388 (54.37%)51 (31.07%)24 (14.56%)
Individualized628400 (63.66%)103 (16.34%)125 (20.00%)
All Buckets2,7351,957 (71.55%)455 (16.64%)323 (11.81%)
Safer Variant (all models)9,1496,760 (73.89%)1,694 (18.52%)695 (7.59%)

The results reveal distinct failure patterns across recommendation policies. Reject failures are predominantly unsafe and incomplete (91.73%), consistent with the relatively explicit requirement to refuse the requested food. In safer-variant cases, the full eight-model analysis shows that 73.89% of conclusively classified failures are unsafe and incomplete, while another 18.52% are unsafe despite satisfying the required recommendation actions. Only 7.59% are safe but incomplete. Thus, among conclusively classified failures, missing safer alternatives alone cannot explain the particularly low strict pass rates.

The more context-dependent policies exhibit different patterns. Clarify-first has a relatively high proportion of unsafe but complete responses (23.30%), indicating that performing the expected recommendation action does not necessarily establish that the resulting permission is safe. This pattern is even more pronounced for conditional recommendations (31.07%), where the correctness of the stated conditions matters. In contrast, individualized recommendations have the highest proportion of safe but incomplete responses (20.00%), reflecting the additional difficulty of identifying and communicating patient-specific decision requirements.

We will clarify this in the future version and report the separate safety and completeness analyses alongside the original Policy Pass results.

AuthorsOpenReview ↗

Response to Reviewer CfGv (2/2)

2. Could a simpler simulator achieve similar results, and what does each DSLR component contribute?

We thank the reviewer for this suggestion. To directly address this question, we conducted an additional comparison of four user simulators on 200 matched scenarios, sharing the same initial exchanges, target and simulator models, and budget of four assistant replies. We evaluated 577 follow-up user turns per condition along two dimensions: whether the follow-up challenged the recommendation boundary and whether it introduced unsupported facts that could change the appropriate recommendation.

User simulatorBoundary-challenging follow-upsUnsupported decision-changing facts
A. Ordinary LLM simulator1293
B. Simple targeted simulator3497
C. Targeted simulator + pressure plan43953
D. Full DSLR5134

First, targeted pressure substantially improves boundary testing. Simply instructing an ordinary LLM simulator to continue pursuing the requested food increases boundary-challenging follow-ups from 129 to 349. Adding a scenario-specific pressure plan further increases this number to 439, but also increases unsupported decision-changing facts from 7 to 53. This shows that generating more pressure does not necessarily produce more valid evaluation conversations.

Second, full DSLR achieves a better balance between effective pressure and factual validity. Compared with the plan-guided simulator, full DSLR generates more boundary-challenging follow-ups (513 vs. 439) while substantially reducing unsupported decision-changing facts (4 vs. 53). Notably, its unsupported-fact count remains comparable to that of the ordinary simulator (4 vs. 3), despite generating substantially more targeted challenges.

Third, our original component ablations provide complementary evidence that DSLR's components affect the difficulty of the resulting evaluation. As reported in the submitted paper, on a separate 200-scenario GPT-5.5 evaluation, full DSLR achieved 56.5% all-turns and 70.5% final Policy Pass. Removing stagewise references increased these scores to 62.0% and 74.5%, while removing response-state guidance increased them to 59.5% and 73.0%, respectively. These results suggest that the components help expose additional strict-policy failures

The components serve complementary roles: the pressure plan specifies the recommendation boundaries to test; the router identifies the assistant's response type; the stagewise references provide scenario-grounded follow-up anchors; and the response-state table guides subsequent challenges based on the interaction history. The new comparison evaluates their combined contribution, whereas the original ablations examine the effects of removing specific guidance components.

Together, these experiments support DSLR's value in generating policy-relevant, factually grounded multi-turn pressure. The contribution is not simply stronger user insistence, but a structured mechanism for challenging recommendation boundaries without introducing unsupported facts that could invalidate the evaluation.

3. Presentation

We thank the reviewer for the constructive suggestions. We will improve the clarity and presentation of the manuscript, particularly Figures 1 and 2, in the revised version.

Thank you for the detailed rebuttal. I think you've done a really good job addressing my questions, and I appreciate the additional experiments.

My biggest concern now is the writing. I had a very hard time following the main paper. There are too many terms, and the explanations are much more complicated than they need to be. I think this is partly why I raised those questions in the first place. Your rebuttal actually explains the ideas much more clearly than the paper does.

I don't have major concerns about the experiments anymore. I still lean toward weak reject because I think the paper needs substantial rewriting to make the main ideas easy to understand. This goes beyond what I would expect from a camera-ready revision.

The work itself is valuable, and a clearer presentation would make a real difference.

We have uploaded a revised PDF that rewrites the presentation, and the clearer explanations from our rebuttal are now part of the main text. Thank you for the follow-up, for recognizing the value of the work, and for pointing out where the original presentation was hard to follow; we agree that it was more complicated than necessary. You noted that the needed rewriting went beyond what you would expect from a camera-ready revision, so we have carried it out now, and it can be judged in the revised PDF itself.

To be specific, the revision addresses the three issues you raised:

1. Too many terms and symbols. Sections 3.2–3.3 now describe dialogue construction in words, without the symbols the original version introduced for it; notation is kept only for truth cards and scoring. Section 3.4 defines final over-permission and stance reversal in a single sentence and uses Figure 1 to illustrate the two scoring views, and a new glossary explains the key terms and symbols in plain language (Appendix Q).

2. Explanations more complicated than needed. We rewrote the description of DSLR in the Introduction and in Sections 3.2–3.3 in plain language: what is fixed before any model is tested (a pressure plan, a reference library of example user turns, and a shared response-state table), and what adapts during the dialogue (the simulator's next user turn). The two analyses from our rebuttal are now in the main text: the separation of unsafe advice from safe but incomplete responses (Section 4.2, full table in Appendix K.5), and the comparison with simpler user simulators (Section 4.3, full table in Appendix N.4).

3. Figures 1 and 2. Figure 1 is now a full-width example, with the dialogue on the left and the two evaluation views on the right. Figure 2 was redrawn as four columns that map one-to-one to Sections 3.1–3.4 and follow a single running example, the same case as in Figure 1. A new Figure 3 shows the data flow of one DSLR round, making explicit what the target model sees and what only the simulator uses.

The revised PDF also includes the additional experiments and analyses from the rebuttal, and the main text still ends on page 9. We would be grateful if you could look at the revised PDF and let us know whether it resolves your remaining concern; we would welcome any further feedback on the presentation.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Response to Reviewer 5Kbk

We thank the reviewer for recognizing the importance of evaluating food recommendation policies across multi-turn interactions and for identifying two areas where additional evidence would improve the paper.

1. Reliability of policy-bucket assignments

We now provide an additional detailed comparison against the original gold policy buckets on 200 scenario records, stratified by their original evidence status.

Evidence statusScenariosReviewer A vs. GoldReviewer B vs. Gold
Strong8074/80 (92.50%)71/80 (88.75%)
Partial114101/114 (88.60%)99/114 (86.84%)
Expert needed66/6 (100%)6/6 (100%)
Overall200181/200 (90.50%)176/200 (88.00%)

Among the 114 partial-support scenarios, Audit A and Audit B match the original gold policy buckets in 101 cases (88.60%) and 99 cases (86.84%), respectively. The six expert-needed entries have unchanged labels in both reviewers' versions.

2. Model rankings under matched dialogue length and pressure points

We analyzed the eligible four-turn dialogues from the full 1,026-scenario archive, corresponding to the maximum number of assistant replies in our original protocol.

For each of the 28 model pairs, we compare the same scenarios under the same prompt condition. Both models must have exactly four assistant replies and the same nonempty set of must-hit point IDs identified before the final reply. Unanswered trailing user messages are excluded. The shared point set may comprise an identical subset of the planned must-hit points.

Each model pair is evaluated on its common matched scenarios. We compute the difference in all-turns pass rates within each pair, then average these differences equally over the seven opponents to obtain the matched model ordering.

Prompt conditionMatched scenarios per model pairDirect pairwise sign agreementKendall τ-b: aggregate matched vs. full-pool ranking
Baseline123–30424/280.929
Explicit Boundary162–27121/280.473

The Baseline matched ordering differs from the original ordering only in the relative positions of DeepSeek and Gemini. Under Explicit Boundary, GPT-5.5, DeepSeek, and Gemini occupy the first three positions, while the ordering of the remaining models changes more substantially.

These results show that much of the Baseline ordering is preserved in the four-turn matched analysis, with weaker ordering agreement under Explicit Boundary. The comparisons are conditioned on naturally occurring four-turn dialogues.

The matched four-turn dialogues also continue to exhibit earlier failures that are hidden by final-answer evaluation. Among unique model–scenario dialogues participating in at least one matched pair, 824/3,973 Baseline dialogues and 783/3,954 Explicit Boundary dialogues passed at the final turn after an earlier policy failure. Each dialogue is counted once in these totals. Consequently, final-answer pass rates exceed all-turns pass rates by 20.74 and 19.80 percentage points, respectively. This provides additional evidence for evaluating policy preservation throughout the dialogue.

Summary of Rebuttal and Revisions

We have uploaded a revised manuscript that incorporates all analyses from the rebuttal and substantially rewrites the presentation. We thank the reviewers and the AC for their time and constructive feedback. The reviewers recognized the practical value of the benchmark and its validation; below we summarize how the revision addresses their five main concerns. In their follow-up, Reviewer CfGv said that they no longer had major concerns about the experiments and identified the writing as their main remaining concern, which is the focus of the presentation rewrite described at the end.

1. Unsafe advice vs. safe but incomplete responses (CfGv). We classified most failed replies as unsafe rather than merely incomplete. Among 2,735 failed Baseline replies from GPT-5.5 and DeepSeek-V4-Pro across all five policies, 71.55% were classified as unsafe and incomplete, 16.64% as unsafe but complete, and 11.81% as safe but incomplete. Among 9,149 failed Safer Variant replies from all eight models under both prompts, 7.60% were classified as safe but incomplete. (Section 4.2; Appendix K.5)

2. Added value of DSLR over simpler user simulators (CfGv). On 200 matched scenarios with 577 follow-up turns per simulator, full DSLR produced 513 boundary-challenging follow-ups and 4 follow-ups with unsupported decision-changing facts, compared with 129 and 3 for an ordinary LLM simulator, 349 and 7 for a simple targeted simulator, and 439 and 53 for a targeted simulator with a pressure plan. (Section 4.3; Appendix N.4)

3. Positioning relative to SYCON Bench and SafeDialBench (7AuU). SYCON Bench studies sycophantic conformity and stance changes under user pressure, and SafeDialBench studies safety under jailbreak attacks. FoodBoundaryBench instead scores the continued fulfillment of scenario-specific, source-grounded recommendation policies, such as offering an accepted safer variant or clarifying missing food details before permitting the food. The per-policy breakdown in Appendix K.5 shows what each policy category diagnoses beyond an aggregate pass rate. (Section 5; Appendix M)

4. Model coverage and intervention analysis (7AuU). On a shared 200-scenario subset, we evaluated four open-weight models (9B–31B) and a later release, GPT-6.1 Sol, and compared model sizes and versions; under Baseline, final-answer pass exceeded all-turns pass for all five models. For GPT-5.5, all-turns pass rose from 47.0% to 53.6% in a full-benchmark analysis of archived BM25 trajectories, and from 38.5% to 49.0% in a separate 400-scenario self-check study. Across the original eight models, Initial Pass was 65.2% under Baseline and 64.0% under Explicit Boundary, while subsequent failure fell from 41.6% to 27.7% among the 4,178 model–scenario pairs that passed initially under both prompts. We also report effects by policy category and break down the net gains of Explicit Boundary and of the original 200-scenario self-check into improvements and regressions. (Section 4.2; Appendices O.2–O.6)

5. Reliability of policy-bucket labels and model rankings (5Kbk). Two reviewers match the gold policy buckets on 90.50% and 88.00% of 200 scenarios, including 88.60% and 86.84% of the partial-support scenarios. In pairwise comparisons matched on four assistant replies and the same nonempty set of raised must-hit points, the Baseline ordering largely agrees with the full-pool ordering (Kendall τ-b = 0.929; 24/28 pairwise signs agree), with weaker agreement under Explicit Boundary (0.473; 21/28). On these matched dialogues, final-answer pass exceeds all-turns pass by 20.74% under Baseline and 19.80% under Explicit Boundary. (Sections 4.2–4.3; Appendices A.4 and P.3)

Presentation (CfGv). We simplified the method description and now describe dialogue construction in words, without the symbols the original version used for it. Section 3.4 adds brief definitions of final over-permission and stance reversal, and Appendix Q adds a plain-language glossary. Figures 1 and 2 were redrawn: Figure 2 now maps each stage of the pipeline to Sections 3.1–3.4 and follows one running example, and a new Figure 3 shows the data flow of one DSLR round. The main text still ends on page 9.

We hope that these analyses and the revised manuscript address the reviewers' concerns, and we thank the reviewers and the AC again.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.