- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces FoodBoundaryBench, a benchmark for evaluating whether conversational LLMs select and maintain appropriate health-conditioned food recommendation policies across multiple turns. It contains 1,026 scenarios grounded in official guidance and organized into five policy categories. The proposed DSLR protocol fixes scenario-level pressure objectives while allowing simulated user follow-ups to adapt to model responses. Evaluation combines structured response extraction with deterministic policy adjudication and reports both final-answer and all-turns compliance. Experiments across eight models and two prompting conditions show that a correct final answer can conceal earlier policy violations. Human audits and sensitivity analyses provide additional validation of this finding.
Critical strengths
-
A useful domain-specific evaluation resource. The benchmark grounds its scenarios in official guidance and distinguishes five appropriate recommendation policies, including clarification, safer alternatives, and conditional advice. This provides a more detailed assessment of food recommendation behavior than a binary refusal-based evaluation and gives the dataset practical value for studying health-aware conversational assistance.
-
Meaningful validation of the empirical findings. The paper includes human audits, evaluator checks, and sensitivity analyses. In particular, human annotators reproduce a final-versus-all-turns compliance gap similar to the automatic estimate. This strengthens confidence that the reported earlier-turn violations reflect identifiable model behavior.
Critical weaknesses
-
The incremental contribution over existing multi-turn safety evaluation is not sufficiently established. Prior benchmarks already assess whether models maintain appropriate behavior under conversational pressure. SYCON Bench evaluates expected behavior at each turn, while SafeDialBench uses the lowest turn-level safety score to assess a dialogue. The official-guidance grounding and five food-specific policy categories are useful refinements, but the paper does not clearly establish what additional diagnostic value they provide beyond existing approaches. This limits the significance of the contribution, particularly because the benchmark and evaluation protocol constitute the paper’s main contributions.
-
The empirical coverage and intervention analysis provide limited generalizable insight. The evaluation concentrates on eight flagship model services, without representative small open-weight models or comparisons across model scales. The selection of model versions could also be better justified given the availability of subsequent releases. Although the paper includes explicit-boundary prompting, retrieval, and self-checking baselines, the supplementary retrieval and self-checking experiments cover only 200 scenarios and one or two target models. The aggregate improvements do not adequately explain which policy categories benefit, whether the gains reflect better initial policy selection or subsequent policy preservation, or why intervention effectiveness differs across models. These limitations reduce the benchmark’s informativeness about the sources of failure and promising directions for improvement.
Questions for authors
Please address the clarification points raised in the weaknesses above, particularly the benchmark’s incremental diagnostic value over prior work, the rationale for model selection, and what the existing intervention results reveal about initial policy selection versus subsequent policy preservation.
4 comments
Response to Reviewer 7AuU (2/4)
2. Expanded empirical coverage and intervention analysis
We have expanded the evaluation and organized the additional analyses around the distinct issues raised in the review. Throughout, Initial Pass refers to the first eligible response, Final Pass to the final eligible response, and All-turns Pass to the complete scored trajectory. Percentage-point changes compare conditions on the same scenarios.
2.1. Open-weight models on a shared 200-scenario subset
We added Qwen3.5-9B, Qwen3.5-27B, Qwen3.8-27B, and Gemma4-31B-it, extending the evaluation to a smaller 9B model, a same-generation 9B/27B comparison, a subsequent-generation 27B model, and another open-weight model family.
Given the rebuttal time constraints, we prioritized a comparison on the same fixed 200-scenario subset used in the paper's supplementary intervention experiments. This subset covers all five policy categories. Reusing it preserves scenario-level comparability across models and aligns the Baseline results with the additional Explicit Boundary evaluation below.
All four models yielded 200 valid Baseline results:
| Model | Initial Pass | Final Pass | All-turns Pass |
|---|---|---|---|
| Qwen3.5-9B | 63.0% | 54.5% | 32.0% |
| Qwen3.5-27B | 67.5% | 53.5% | 34.0% |
| Qwen3.8-27B | 67.0% | 56.0% | 35.5% |
| Gemma4-31B-it | 64.5% | 53.0% | 28.5% |
These results extend the model coverage beyond the original flagship services and show that the gap between initial, final, and whole-dialogue performance also occurs in the added open-weight models.
2.2. Comparisons across model sizes and versions
Given the limited time available for the rebuttal, we focus this additional analysis on two complementary comparisons using the same fixed 200-scenario subset as in the paper's supplementary experiments. Reusing this subset preserves scenario-level comparability across model sizes and versions. All comparisons use the Baseline condition and a shared auxiliary-model configuration. Qwen3.5-9B versus Qwen3.5-27B provides a same-generation size comparison, while Qwen3.5-27B versus Qwen3.8-27B provides a version comparison at the same nominal parameter scale.
| Comparison | Initial Pass difference | Final Pass difference | All-turns Pass difference |
|---|---|---|---|
| Qwen3.5-9B → Qwen3.5-27B | +4.5 | −1.0 | +2.0 |
| Qwen3.5-27B → Qwen3.8-27B | −0.5 | +2.5 | +1.5 |
In the size comparison, the 27B model has a higher observed Initial Pass, while the All-turns difference is smaller. In the version comparison, Initial Pass is similar (67.5% versus 67.0%), while Qwen3.8-27B has slightly higher observed Final Pass (56.0% versus 53.5%) and All-turns Pass (35.5% versus 34.0%). These results extend the evaluation across both model sizes and versions while distinguishing initial, final, and whole-dialogue performance.
2.3. Model versions and evaluation on a subsequent release
To address the availability of subsequent releases directly, we added GPT-6.1 Sol, evaluating Baseline and Explicit Boundary on the same fixed 200 scenarios. All 200 results in each condition were valid.
| Metric | Baseline | Explicit Boundary | Change |
|---|---|---|---|
| Initial Pass | 70.0% | 71.5% | +1.5 |
| Final Pass | 50.0% | 64.0% | +14.0 |
| All-turns Pass | 35.0% | 53.5% | +18.5 |
The newly evaluated release continues to exhibit a substantial difference between final-answer success and success throughout the interaction, while benefiting from Explicit Boundary.
The original eight-model cohort was selected to provide broad coverage across model families during the main evaluation of 21–24 May 2026.
Response to Reviewer 7AuU (3/4)
2.4. Expanded retrieval and self-check evaluation, with an explicit sampling rationale
Retrieval. We expanded the analysis of archived GPT-5.5 BM25 trajectories from the original 200-scenario appendix subset to the full benchmark. All-turns Pass increases from 47.0% to 53.6%, a gain of 6.7 points. Among the usable scenarios outside the original 200 subset, the gain is 7.2 points. This directly checks whether the original-subset finding extends to the remainder of the benchmark.
Self-check. We completed a new GPT-5.5 study on 400 distinct scenarios, comprising the original 200 and a pre-fixed, non-overlapping extension of 200.
| Self-check cohort | Baseline → Self-check All-turns Pass | Gain |
|---|---|---|
| Original subset | 40.0% → 48.5% | +8.5 |
| Non-overlapping extension | 37.0% → 49.5% | +12.5 |
| Combined designed sample | 38.5% → 49.0% | +10.5 |
The extension was sampled with a fixed random seed from the 826 scenarios outside the original subset, with 40 scenarios per policy category. Given the rebuttal time budget, we prioritized completing this matched extension to test the findings on additional scenarios while preserving coverage of every policy category. Both conditions use the same target and auxiliary-model configuration, and the simulator sees the revised response in Self-check.
2.5. Initial Pass versus subsequent preservation
We now examine these stages for Explicit Boundary, retrieval, and self-check.
Across the original eight models and 1,026 scenarios, Explicit Boundary changes aggregate Initial Pass from 65.2% to 64.0%, while All-turns Pass increases from 35.3% to 45.0%. Among the 4,178 model–scenario pairs that pass initially under both conditions, subsequent failure decreases from 41.6% to 27.7%. This supplies evidence about preservation after an initially satisfactory response.
The added open-weight experiments show the same distinction. Initial Pass changes by −3.5, +1.5, +1.5, and −3.0 percentage points, respectively. To examine subsequent policy preservation independently of differences in initial pass status, we focus on scenarios where the model achieves Initial Pass under both Baseline and Explicit Boundary. Among these shared initially passing scenarios, subsequent failure rates decrease from 41.1% to 30.0%, 43.2% to 27.9%, 43.4% to 23.6%, and 45.7% to 33.0%, respectively. For GPT-6.1 Sol, the same analysis covers 126 scenarios, with subsequent failure decreasing from 46.8% to 23.8%.
Retrieval has a different observed profile. For GPT-5.5 BM25, Initial Pass increases from 65.0% to 71.5%; among the 605 scenarios passing initially under both conditions, subsequent failure decreases from 24.8% to 19.8%. In the new 400-scenario Self-check study, Initial Pass changes by only +1.5 points, while Final Pass and All-turns Pass improve by 11.5 and 10.5 points.
Response to Reviewer 7AuU (4/4)
2.6. Which policy categories benefit, and where trade-offs remain
For Explicit Boundary across the original eight models, All-turns Pass gains are 19.4 points for Reject, 15.0 for Safer Variant, 18.6 for Clarify, and 13.5 for Conditional, while Individualized decreases by 7.8 points. Initial Pass in Individualized decreases by 24.2 points, helping explain why the overall Initial Pass does not improve despite gains in several other categories.
The new open-weight results provide a concrete preservation example. For Qwen3.5-27B in Clarify, Initial Pass remains 89.6%, but All-turns Pass increases from 47.9% to 77.1%. Among the 39 scenarios passing initially under both conditions, subsequent failure decreases from 46.2% to 15.4%.
They also reveal trade-offs: Individualized Initial Pass decreases from 77.6% to 56.9% for Qwen3.5-9B and from 91.4% to 55.2% for Gemma4-31B-it. All-turns Pass changes are mixed across the four added open-weight models, showing no consistent improvement in this category. This effect is not universal across targets: GPT-6.1 Sol's Individualized All-turns Pass increases from 44.8% to 74.1%.
For retrieval, Clarify has the largest observed full-benchmark BM25 All-turns gain, 12.6 points; the Conditional is −1.8 points. In the independent 200-scenario Self-check extension, Reject and Safer Variant improve by 27.5 and 20.0 points, while the Conditional is −10.0 points.
These results identify which recommendation requirements improve, where initial execution becomes harder, and where the direction of the effect depends on the model or intervention.
2.7. Why observed intervention gains differ across models
We decompose net All-turns Pass gains into Fail → Pass improvements and Pass → Fail regressions. Both percentages use all paired scenarios for the corresponding model as the denominator.
This analysis covers all eight original models. For example, under Explicit Boundary, GPT-5.5 has 13.3% improvements and 9.2% regressions; Gemini 3.1 Pro has 19.7% improvements and 16.0% regressions; and Kimi K2.6 has 25.8% improvements and 10.4% regressions, each over 1,026 scenarios. Thus, a small net gain can reflect relatively few improvements or a larger offset from regressions.
The same analysis extends to all five newly added Explicit Boundary targets:
| Model | Paired scenarios | Fail → Pass | Pass → Fail | Net All-turns Pass gain |
|---|---|---|---|---|
| Qwen3.5-9B | 200 | 19.0% | 11.0% | +8.0 |
| Qwen3.5-27B | 200 | 23.5% | 9.5% | +14.0 |
| Qwen3.8-27B | 200 | 22.0% | 10.0% | +12.0 |
| Gemma4-31B-it | 200 | 22.0% | 9.0% | +13.0 |
| GPT-6.1 Sol | 200 | 27.0% | 8.5% | +18.5 |
Improvements and regressions coexist in every added target. For example, Qwen3.5-27B's larger observed gain relative to Qwen3.5-9B combines a larger improvement share with a smaller regression share.
The self-check results provide another example: GPT-5.5 has 17.5% improvements and 6.5% regressions, whereas DeepSeek-V4-Pro has 16.0% and 14.5%, respectively, each over the same 200 scenarios. Their different net gains, 11.0 versus 1.5 points, therefore accompany similar improvement shares but substantially different regression shares.
This decomposition explains the outcome changes behind the aggregate effects under the evaluated configurations. Together with the stage and category analyses, it supports evaluating interventions for both their ability to satisfy previously unmet recommendation requirements and their ability to preserve requirements already satisfied.
Response to Reviewer 7AuU (1/4)
We thank the reviewer for recognizing the value of the source-grounded recommendation policies and the validation of earlier-turn failures. We address the two concerns below: the additional diagnostic information provided by the benchmark, and the breadth and interpretation of the empirical results.
1. Incremental diagnostic value beyond existing multi-turn safety evaluation
FoodBoundaryBench was designed to evaluate safe food recommendation in multi-turn interactions. Success in this setting requires both respecting source-grounded food–health safety constraints and completing the recommendation actions required by the scenario. Both requirements are already included in our original task definition and Policy Pass criterion (Section 2).
SYCON Bench examines sycophantic conformity and stance changes under user pressure, while SafeDialBench evaluates unsafe-information identification, handling, and safety consistency under jailbreak attacks. FoodBoundaryBench focuses on the continued fulfillment of scenario-specific, source-grounded recommendation policies. These policies specify actions such as providing an acceptable safer variant, clarifying missing food details before giving permission, or preserving the conditions governing a recommendation. Appendix C already illustrates how a response can preserve a safety stance while remaining incomplete under the required recommendation policy.
To make this original evaluation scope and its diagnostic value clearer, we added a separate analysis of safety and recommendation-action completeness to the existing Policy Fail results. We distinguish unsafe and incomplete responses, unsafe but complete responses, and safe but incomplete responses, and report their proportions across the five policy categories. This additional analysis makes explicit which components of the original task requirements are unmet and how the failure patterns differ across policies.
The results reveal distinct failure patterns. Reject failures predominantly involve both unsafe advice and incomplete recommendation actions (91.69%). For Safer Variant, the eight-model analysis assigns 73.89% of failures to unsafe and incomplete responses and 18.52% to unsafe responses despite complete recommendation actions; 7.60% are safe but incomplete. Thus, otherwise-safe responses with incomplete actions do not account for most of the observed Safer Variant failures.
The separation is particularly informative for the more context-dependent policies. Unsafe but complete responses account for 23.36% of Clarify-first failures and 31.29% of Conditional failures. These cases illustrate why carrying out the expected recommendation action does not by itself establish the safety of the resulting advice. Conversely, Individualized has the largest safe but incomplete share (19.90%), identifying failures in which avoiding unsafe advice and fully satisfying the required recommendation actions come apart.
This decomposition makes the diagnostic role of the policy categories explicit: it identifies both the safety status of a failed response and the recommendation requirement involved. It distinguishes unsafe advice that also omits required actions, unsafe advice despite complete actions, and otherwise-safe responses with incomplete actions. These distinctions provide more specific information about what needs improvement than an aggregate Policy Pass rate alone.