- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether the advantage of hidden-state-based diagnostic scores over answer loss transfers in soft-prompt tuning, where the task is to predict which questions change correctness status between two prompts.
It reports three main findings. Firstly, in two Ministral–ECQA prompt pairs, the state scores' advantage comes from whether outputs pass strict parsing, and the ranking reverses once outputs are constrained. Secondly, a crossed design shows that this difference varies mainly across prompt groups and little across question sets. Lastly, after charging measurement costs, direct checks are more efficient than screening with the diagnostic scores. The experiments are rigorous and honestly reported. However, in independent validation the proposed diagnostic underperforms simple baselines such as answer loss, JS divergence and a linear probe, and the transfer and cost conclusions are also negative.
The paper has no positive result that supports the value of the framework, so I lean toward a weak reject, rating 2.
Critical strengths
This paper workflow and presentation is relatively complete and professional. 1.The experimental procedure is rigorous. Predictions are locked before the confirmation data are generated, the evaluation uses fresh questions and independently trained prompts, multiple comparisons are Bonferroni-corrected, and undefined AUCs are never filled in. 2. Negative results are reported honestly. There is few overclaim in paper. 3. The constrained-output intervention in section 3 is cleanly designed. The diagnostic scores and the first answer letter are unchanged and only the continuation is removed, yet the state-minus-loss gap changes by −0.672 and −0.845. This shows that a diagnostic's apparent advantage can come from the output format.
Critical weaknesses
There are several weakness. From my perspective, I hope you can polish this work carefully.
Firstly, the proposed diagnostic underperforms simple baselines. In table 1, the state AUCs are 0.460, 0.549 and 0.622, below chance on Ministral. Answer loss reaches 0.834, 0.731 and 0.895. JS divergence reaches 0.850, 0.761 and 0.885. An ordinary linear probe also exceeds CF-PED on all three models.
Secondly, the control-response components of the method contribute thin. The norm-only control and the random-subspace control perform about the same as the full state/readout score, like parseability-channel AUCs of 0.785 and 0.791 versus 0.787 on the discovery set. The selected rank is 32, the upper limit of the candidate set, and the held-out residual energy is still 0.55–0.90. The gradient and subspace features therefore show no measurable effect, and the need for a "control-response framework" isn't supported.
Finally, the transfer and practical-value conclusions are all negative. The improvement of pair-specific forecasts over the uniform mean has intervals that include zero. On GSM8K, the full forecast's MAE (0.097) is worse than predicting zero (0.087). In the cost analysis, the state scores never beat direct inspection in any of the 54 pairs. The gain of the prompt update in Section 6.1 over a random objective also has an interval that includes zero. The paper has no positive result that supports the value of your proposed framework.
I hope authors can give a detailed explanation. If you clarify these questions, I will update my score.
Questions for authors
Please see the weakness section. Thank you for your efforts.
Thank you for your careful comparison of CF-PED with simpler baselines and for asking which improvements matter in practice.
The submitted CF-PED and its baselines. On independent ECQA pairs, the submitted diagnostic scored 0.460/0.549/0.622 AUC on Ministral/Granite/Qwen. Loss reached 0.834/0.731/0.895 and JS divergence 0.850/0.761/0.885. The linear probe also beat CF-PED on all three (Table 1). These results make loss and JS the first benchmarks for a state score intended to find questions whose credit will change.
What can be improved? Swapping prompts leaves the change label identical. The original features included signed terms that depend on endpoint order. We compared ordered and swap-invariant features with local and integrated gradients, selected on development data, and froze the rule. On 32 endpoint-disjoint pairs per cell, the table gives the equal-pair mean AUC gain over a rank-32, local ordered refit of the CF-PED structure.
† Supplementary confirmations (Table 4). Intervals use 20,000 shared, crossed question/pair bootstrap draws and correction over all 96 comparisons. The symmetric combination improved ranking over the refit in all three ARC families and on ECQA. BoolQ's interval includes zero. On ARC–Ministral it also beat exact margin and the linear probe after correction (Appendix R, Table 37). Loss and JS still had higher mean AUC in all five cells.
Which design choice mattered? At fixed selected rank, changing local ordered to local symmetric features raised mean AUC from .805 to .902, .590 to .701, .726 to .848, .661 to .766 and .888 to .926, in table order. Switching the symmetric features from local to integrated gradients changed mean AUC by at most .004. These component comparisons are descriptive. The corrected gains above apply to the whole rule. In the original discovery set, parseability AUC was .787 for CF-PED, .785 for the norm alone and .791 for a random subspace. The original rank reached its ceiling of 32, with residual fractions .5545 (Granite–ECQA), .8264 (Ministral–ECQA) and .8998 (Ministral–GSM8K). All five new cells reached the expanded ceiling of 256, with residuals .285–.543. On BoolQ the rank-matched random basis scored .927 versus .926 for the selected rule. For a developer, the local, swap-invariant score is therefore a useful baseline for testing what integration or fitting a basis adds at the same rank and data budget.
What transfers? Forecasts locked before the third panel had lower point MAE than the common mean in all five cells. A known pair's previous state-minus-loss AUC gap can thus inform its forecast on fresh questions. Historical improvement intervals over the common mean span zero. For unseen GSM8K prompts, the full forecast had MAE .097 versus .087 for predicting a zero gap. Gains from the prompt-update objective over random were 1.13 [−5.75, 8.00] and 1.31 [−1.25, 4.88] percentage points. Both gains and losses are compatible with these intervals. Choosing a training objective needs its own test.
When does ranking pay for itself? The submission replayed costs on 54 pairs. We now timed four full strategies on two prespecified pairs per cell using panel 0. Each entry gives the changes found by direct checking on the first/second pair at three calibrated budgets (Appendix S).
At 10% and 25%, screening exhausted the budget before completing any check under both prompts. At 50%, ARC–Granite loss found 0/2 changes. ARC–Qwen loss, JS and the selected score found 20/11, 15/9 and 11/6. Direct checking matched or beat every screen in all 30 observed pair–budget conditions. On ARC–Qwen, even the strongest screen tied direct checking on both pairs at 50%.
The historical Ministral–ECQA replay explains the trade-off. At the reference budget, loss screening's higher hit rate added 20.51 discoveries per pair on average, while fewer completed checks cost 43.56. An evaluator can measure both terms under their own time budget to see whether screening pays (Eq. 7).
A separate supplementary GSM8K preflight timed one historical pair on six calibration questions. Direct checking took 31.83 s and loss screening 33.15 s. Both checked all six without a per-policy deadline and each found one correctness change. Outputs were 23–111 tokens under a 512-token cap. These are full-pass timings. The corresponding long-answer decision requires comparing changes found in equal time.