- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This work studies PECF, where LLMs give directional advice grounded in frameworks lacking established predictive validity, such as BaZi fortune-telling. The motivation is to separate Advice Strength (A) from Grounding Source (G), because the same recommendation can rest on either evidence-based reasoning or pseudo-expert premises, and users need to know the stated basis of consequential advice. The experimental method uses four commercial models across 11,512 explicit BaZi-request responses and a paired carryover probe, scoring responses on A and G axes where levels ≥2 define the joint PECF metric. Results show explicit BaZi requests yield a high PECF while grounding carries over into follow-up questions that omit framework keywords, and system-prompt interventions separate advice frequency from grounding with strongly model-dependent effects.
Critical strengths
-
[S1] The paper studies an interesting problem: how an initial BaZi or Tarot scenario shapes the model’s responses to subsequent follow-up questions even when the follow-up omits the "BaZi" or "Tarot" keywords. The experimental results show that these pseudo-expert grounding persists into the next turn dialogue, indicating that conversational history can affect practical advice beyond the initial prompt.
-
[S2] The paper develops an A/G two-dimension rubric that scores Advice Strength (A) and Grounding Source (G) independently, with levels 0 ~ 3 on each dimension and the joint PECF metric defined as A ≥ 2 and G ≥ 2. This separation enables more fine-grained assessment and reveals how system-prompt interventions can change grounding while retaining advice frequency or vice versa.
Critical weaknesses
-
[W1] The paper investigates the tendency of LLMs to provide advice based on pseudo-expert frameworks like BaZi. However, the primary finding that models follow the requested framework when explicitly prompted is not counterintuitive. If the initial prompt explicitly asks the model to act as a traditional fortune-teller and provide a BaZi analysis, it is expected that the model would maintain that role and its associated logic in subsequent turns.
-
[W2] The presentation clarity of the paper needs improvement, and the underlying design intuition and experimental motivation require refinement. For instance, the experimental naming scheme is opaque: conditions are labeled 1C, 3A, 4A–4D, 5A, and 5B, but the paper never explains what the letters and numbers signify, making it difficult to track the design logic across tables and figures. In lines 141 and 143, the definitions for the neutral decision scenario and the fifth domain of Tarot are not clearly established. Section 3.3, the design intuition of the 4 prompts can be further expanded for comprehensiveness. Section 3.4 does not fully motivate why the two-dimensional A/G decomposition is the appropriate basis for evaluating pseudo-expert grounding. The relationship between the rubric and the binary thresholds (A ≥ 2, G ≥ 2) could be explained more transparently as well.
-
[W3] Basically all of the experimental labels are annotated by LLM judges (GPT-5.4 & MiMo-V2-Pro). The human-labeled subset comprises only 128 responses, which represents approximately 1% of the total dataset (11,512). While collecting human annotations is resource-intensive and costly, this sample size is too small to robustly validate the technical soundness of the results or support the claim in Section 4.4 that automated assessment maintains high agreement with human judgment.
Questions for authors
- [Q1] The paper investigates PECF using BaZi and Tarot frameworks as contexts. However, the connection between these findings and the broader themes discussed in the literature is not fully explored. Specifically, how do the authors clarify how the observed PECF patterns relate to or extend existing research on safety measurement and sycophancy as mentioned in the related work section? The evaluation could be strengthened if the authors integrated more diverse or relevant scenarios to better demonstrate the practical risks of pseudo-expert grounding in LLM suggestion.