- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work PLAIT turns hallucination-detector scores into audit plans under a fixed review budget. It learns corrections from generator hidden states with a plan-regret objective, jointly selects review actions that share opening costs, and constrains how much of the parent-estimated value the plan retains. Experiments cover four generators, parent replacement, domain transfer and human review. PLAIT-C improves fixed-budget recall by 3.54 percentage points over exact score fusion, and improves unsupported characters identified per minute by 12.47% over HLA-RICO.
Critical strengths
The automatic evaluation includes a strong exact-fusion baseline that uses the same actions, costs and solver. PLAIT-C improves recall from .4691 to .5045, with a 95% confidence interval of 1.80, 5.28 percentage points for the gain
The randomized 96-participant crossover study measures actual review outcomes instead of relying only on simulated capture. PLAIT-C increases correctly identified unsupported characters per minute from 129.1 to 145.2 compared to HLA-RICO
The retention constraint is explicitly separated from a truth guarantee. It satisfies the specified 80% bound in every evaluated session, while a mean-matched soft penalty violates the bound in 21.7% of sessions
Critical weaknesses
The provenance of the “original-generator” states is unclear. Table S34 lists RAGognizer checkpoints for the four generator entries without separately identifying the original generation checkpoints. This ambiguity matters because detector-derived states would change the interpretation of the latent-signal contribution.
The BCE comparison changes both the training objective and the checkpoint-selection criterion. Appendix D and Table S20 show that BCE checkpoints are selected by validation BCE rather than audit utility, so the reported 5.92-point advantage does not isolate the effect of plan-regret training.
The human study omits the strongest automatic comparator, validation-selected exact fusion. Appendix J shows that HLA-RICO instead uses pointwise learning and risk-to-cost ranking. The reported 12.47% human improvement therefore combines differences in learning and allocation without establishing an advantage over the strongest matched planner.
Questions for authors
Which exact checkpoints and input templates produced the latent features, and how do they relate to the RAGognizer repositories listed in Table S34?
Does PLAIT keep its advantage when the BCE checkpoints are selected with the same validation audit-utility criterion?
Does the human-review improvement still hold against validation-selected exact fusion using the same shared-cost planner?
Thank the reviewer for their response.
I appreciate the results of DeCCaF in human study, where the advantage of proposed method is still statistically significant.
Therefore, I maintain my support for acceptance.
I do not further raise the score because of the minor concern that the increase in hallucination detection throughput compared to DeCCaF is <10%, though statistically significant.