acceptodds
Desk rejected at ICLR 2027

Forecasting Before the Outcome: Catalogue Gated Prequential Cognitive Performance under Participant and Domain Shift

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Online assistance needs a performance forecast before the response and a principled way to decide whether source knowledge transfers to the current participant. We introduce the Support-Aware Prequential Forecaster (\method), which uses a fixed catalogue of supported experiment identifiers to retain source predictions when transfer is expected and switches to hierarchical Beta adaptation for unseen tasks. Every route decision and probability is committed before the current response; feedback updates only subsequent trials. On an external auditory cohort with 13 participants and 28,452 trials, SAPF achieves NLL 0.066 and AUROC 0.929, while immediate source exclusion increases first-five-trial Brier by 0.039 relative to retaining the source. A feedback-weighted SafeMix baseline is useful when catalogue membership is misleading, reducing macro Brier from 0.113 to 0.071 under identifier conflict. Two-expert and temporal ablations show that the main gain comes from the route decision and from recovering source weight after transient failures. In a prefix-causal replay with a 10% budget cap, SAPF covers 42.2% of errors at a 5.8% call rate, compared with 9.0% coverage at a 9.5% call rate for random allocation. These results position pre-response forecasting as an explicit transfer-control layer: retain source knowledge when catalogue support holds, fall back at unsupported cold starts, and update only when later feedback justifies a change. Together, the results show how SAPF turns transfer uncertainty into an actionable routing decision for budgeted online assistance under participant and domain shift.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Prequential forecasting, Cognitive performance prediction, Online adaptation, Distribution shift, Test-time adaptation, Socratic agent

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_F1v5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the Support-Aware Prequential Forecaster (SAPF), which predicts cognitive performance before the current response and selects between a source-trained predictor and online adaptation based on catalogue membership. It also introduces SafeMix, which adjusts expert weights using past predictive losses, and explores budgeted intervention allocation and evidence-grounded Socratic assistance. Experiments across several cognitive-performance datasets show promising forecasting and transfer results, while controlled tests examine robustness to metadata errors and temporal perturbations. The work addresses an important practical problem, although the generality of its routing strategy and the effectiveness of the complete intervention pipeline require further validation.

Critical strengths
  1. The paper clearly formulates the pre-response prediction problem and carefully separates forecasting from retrospective estimation. The prequential evaluation protocol, participant-disjoint splits, and explicit exclusion of current outcomes help address information leakage.
  2. The study includes multiple baselines, external evaluation, routing ablations, and temporal robustness tests. The comparison between hard catalogue-based routing and feedback-based SafeMix is particularly useful for understanding when prior knowledge should be retained or downweighted.
Critical weaknesses

Major concerns

  1. The main external evaluation includes only 13 participants from one auditory cohort. Although the authors appropriately use participant-level bootstrap intervals, the results may not generalize to other tasks, populations, or sensing environments. Moreover, SAPF's participant-balanced Brier score (0.017) is comparable to Context Beta (0.017) and slightly worse than SafeMix (0.016). The advantage is therefore metric-dependent rather than uniformly established.
  2. SAPF relies on exact experiment-identifier matching to determine whether source knowledge should be retained. This is a simple and interpretable rule, but identifier membership does not necessarily indicate actual transferability. The paper's own identifier-conflict experiment shows that SafeMix substantially outperforms exact-ID routing when metadata is misleading. More evidence is needed to establish when the proposed hard-routing strategy is preferable to adaptive alternatives.
  3. Forecasting, error localization, verification, and human assistance are evaluated in largely separate experiments. The assisted-reasoning pilot includes only seven participants, and the verifier-on and verifier-off conditions produce identical reported F1 scores. Consequently, the evidence does not yet establish that integrating these components improves actual task performance or learning outcomes. Minor concerns
  4. The manuscript is lengthy and covers several related but distinct components. A clearer separation between the core forecasting contribution and the auxiliary intervention-agent studies would improve focus.
  5. The paper should clarify the practical interpretation of its different performance metrics and more explicitly distinguish improvements in probability estimation, error ranking, and intervention effectiveness.
Questions for authors
  1. How sensitive is SAPF to errors in the source catalogue? Can the authors identify conditions under which hard routing consistently outperforms SafeMix or other adaptive gating methods?
  2. Given the small external cohort and the comparable participant-balanced Brier scores of SAPF, Context Beta, and SafeMix, how robust are the main conclusions across independent participants, tasks, and domains?
  3. Since the verifier-on and verifier-off conditions yield identical reported F1 scores, what measurable benefit does verification provide? Can the authors demonstrate that the complete forecasting-and-intervention pipeline improves task accuracy or learning outcomes in a sufficiently powered evaluation?
4 comments
AuthorsOpenReview ↗

Response to Reviewer F1v5 (Part 1/4: W1–W4)

Thank you for focusing on when hard routing helps and how its evidence relates to feedback-based weighting. We added stateful routing tests, historical compatibility analysis, and participant-level sensitivity results.

W1: The main external evaluation includes only 13 participants from one auditory cohort. Although the authors appropriately use participant-level bootstrap intervals, the results may not generalize to other tasks, populations, or sensing environments. Moreover, SAPF's participant-balanced Brier score (0.017) is comparable to Context Beta (0.017) and slightly worse than SafeMix (0.016). The advantage is therefore metric-dependent rather than uniformly established.

A1: SAPF's early Brier and full-stream NLL gains over SafeMix persist across all 13 leave-one-participant-out estimates and all 1,716 overlapping six-person subsets. Bidirectional source refits add two task-transfer directions on the 12 held-out Brain participants.

W2: SAPF relies on exact experiment-identifier matching to determine whether source knowledge should be retained. This is a simple and interpretable rule, but identifier membership does not necessarily indicate actual transferability. The paper's own identifier-conflict experiment shows that SafeMix substantially outperforms exact-ID routing when metadata is misleading. More evidence is needed to establish when the proposed hard-routing strategy is preferable to adaptive alternatives.

A2: The matched EWA comparison fixes both source and SAPF fallback forecasts. SAPF reduces first-five Brier by approximately 22% at both learning rates in the unsupported external setting, and remains lower across ten initial-source-prior/learning-rate combinations spanning priors 0.10–0.90. Six source refits characterize source utility across training directions. A historical-compatibility gate estimated from source-training forecasts fixes four task-level routes before evaluating the twelve held-out participants. In a descriptive agreement analysis, the gate selects the lower-loss branch in 19/24 participant-direction cells, compared with 11/24 for exact catalogue membership; these cells reuse the same 12 participants across directions. Stateful perturbations test sensitivity to eligibility errors.

W3: Forecasting, error localization, verification, and human assistance are evaluated in largely separate experiments. The assisted-reasoning pilot includes only seven participants, and the verifier-on and verifier-off conditions produce identical reported F1 scores. Consequently, the evidence does not yet establish that integrating these components improves actual task performance or learning outcomes.

A3: The matched verifier replay removes all 44 action–type incompatibilities from 387 fixed candidates. It retains 343 candidates, with span F1 and item coverage reported alongside the count-matched random control. In the seven-participant assistance-content comparison, the accuracy point estimate is +5.7 percentage points; confidence calibration, helpfulness, and trust also favor item-specific prompts.

W4: The manuscript is lengthy and covers several related but distinct components. A clearer separation between the core forecasting contribution and the auxiliary intervention-agent studies would improve focus.

A4: The revised manuscript centers on pre-response source eligibility, online adaptation, and transfer evaluation. Sections 2 and 3.4 identify the Agent study as a complementary component evaluation. Section 5.6 separates online allocation, localization, and the pilot, while Appendix L contains model settings, type-matched localization, verifier analysis, and pilot details.

AuthorsOpenReview ↗

Response to Reviewer F1v5 (Part 2/4: W5 & Q1)

W5: The paper should clarify the practical interpretation of its different performance metrics and more explicitly distinguish improvements in probability estimation, error ranking, and intervention effectiveness.

A5: The results address three decisions: estimating correctness probabilities, selecting likely errors under a budget, and measuring participant outcomes after assistance. Each has its own metrics and interpretation:

R3-Table1: Prediction, allocation, and assistance endpoints

EndpointMetricsInterpretation
Probability estimationBrier, NLL, ECEBrier measures squared probability error; NLL penalizes confident errors strongly; ECE measures binned calibration
Error retrieval and allocationAUROC, Error AP, error coverage, precision, actual call rateRanking quality and the errors selected by the stated online policy
Assistance outcomeAccuracy, confidence Brier, helpfulness, trustObserved participant outcomes under the compared assistance conditions

For allocation, risk is rt=1−p^tr_t=1-\widehat p_t. A trial is selected when

at=1rt≥τt,1Nt−1<⌊bt⌋.a_t=\mathbf{1}\\{r_t\geq\tau_t\\}\\,\mathbf{1}\\{N_{t-1}<\lfloor bt\rfloor\\}.

where Nt−1N_{t-1} is the number of prior calls, b=0.10b=0.10, and τt\tau_t is a historical risk quantile initialized from development data. Each decision uses prior risks and the current committed prediction, so the policy operates before the response with a budget cap defined for every stream prefix.

R3-Table2: Online allocation under a shared 10% prefix cap (participant means)

Forecast or policyActual call rateError coverage
SAPF + history quantile5.85%42.16%
Context Beta + history quantile6.07%42.63%
SafeMix + history quantile3.07%29.35%
Random allocation9.52%9.01%

At a 5.85% call rate, SAPF covers 42.16% of errors, about 4.7 times the random policy's 9.01% coverage at a 9.52% call rate, and reaches coverage comparable to Context Beta at a lower call rate. The prefix-causal replay shows that pre-response risk can guide budgeted error targeting; the pilot separately measures responses to assistance content.

Q1: How sensitive is SAPF to errors in the source catalogue? Can the authors identify conditions under which hard routing consistently outperforms SafeMix or other adaptive gating methods?

A6: SAPF retains its first-five Brier advantage over SafeMix when eligibility is independently flipped with probability 0.20 at each trial. We reran the full prediction and update sequence under

St(ϵ)=S(ct)⊕Bt,Bt∼Bernoulli⁡(ϵ).S_t^{(\epsilon)}=S(c_t)\oplus B_t,\qquad B_t\sim\operatorname{Bernoulli}(\epsilon).

where ⊕\oplus flips the eligibility bit when Bt=1B_t=1. At each interior rate, five masks are sampled independently of outcomes; 0% and 100% are deterministic endpoints. Source predictions, trial order, hyperparameters, and outcomes are fixed. Because each flip also changes subsequent fallback updates, the replay captures accumulated state effects.

R3-Table3: Stateful trial-level eligibility perturbations on 13 external auditory participants

Trial-level flip probabilityFirst-five BrierSAPF − SafeMix, first five (95% CI)Full-stream Brier
0%0.0991−0.0395 [−0.0540, −0.0177]0.0174
5%0.1046−0.0340 [−0.0489, −0.0133]0.0209
10%0.1109−0.0277 [−0.0427, −0.0089]0.0241
20%0.1159−0.0226 [−0.0366, −0.0058]0.0306
40%0.1319−0.0067 [−0.0198, +0.0074]0.0443
100%0.1668+0.0282 [+0.0177, +0.0352]0.0843

At 20% error, the paired SAPF−SafeMix difference is −0.0226 (95% CI [−0.0366, −0.0058]). Intervals average perturbation seeds within participant and resample the 13 participants; each is pointwise for its flip rate.

Across the six Brain refits, source utility varies by training direction. A separate historical gate, fixed using leave-one-participant-out forecasts from the eight source-training participants, selects the lower-loss branch in all four held-out task-direction comparisons. Its descriptive agreement is 19/24 participant-direction cells, versus 11/24 for exact catalogue membership; the same 12 participants appear in both directions. These results show how historical compatibility can inform eligibility while feedback weighting adapts source influence. Appendix F reports both analyses.

These results define the operating range: the early advantage remains supported at 20% independent eligibility flips, remains directionally favorable at 40%, and complete inversion marks the boundary where the catalogue signal is no longer informative. Appendix F.2 distinguishes these trial-level flips from persistent task-level catalogue errors.

AuthorsOpenReview ↗

Response to Reviewer F1v5 (Part 3/4: Q2)

Q2: Given the small external cohort and the comparable participant-balanced Brier scores of SAPF, Context Beta, and SafeMix, how robust are the main conclusions across independent participants, tasks, and domains?

A7: Every leave-one-participant-out estimate preserves SAPF's first-five Brier and full-stream NLL advantages over SafeMix. The paired gains favor SAPF in 11/13 participants for first-five Brier and 12/13 for full-stream NLL.

R3-Table4: Primary external endpoints on participant-balanced scores

MethodFirst-five BrierFull-stream NLL
SafeMix0.13850.0725
SAPF0.09910.0671

Scores first average within each participant and then across participants, giving each person equal weight. Full-stream NLL is

\left[-y_{it}\log\widehat p_{it}-(1-y_{it})\log(1-\widehat p_{it})\right].$$ SAPF's participant-mean NLL is 0.0671; pooling the same 28,452 trials gives 0.06582, reported as 0.066 in the paper. Participant-mean NLL weights people equally, whereas pooled NLL weights trials equally. Sensitivity analyses use all 13 leave-one-person-out estimates and all 1,716 overlapping six-person subsets. > R3-Table5: Participant sensitivity of SAPF − SafeMix differences | Metric and window | Paired mean (95% CI) | Participants favoring SAPF | Leave-one-out range | |---|---:|---:|---:| | Brier, first five | −0.0395 [−0.0540, −0.0177] | 11/13 | [−0.0462, −0.0383] | | Brier, first 20 | −0.0133 [−0.0189, −0.0071] | 11/13 | [−0.0153, −0.0125] | | NLL, full stream | −0.0054 [−0.0082, −0.0022] | 12/13 | [−0.006620, −0.004602] | Across leave-one-out estimates and overlapping six-person subsets, SAPF preserves its first-five and first-20 Brier and full-stream NLL advantages over SafeMix. SAPF also lowers full-stream NLL by 0.00072 relative to Context Beta (95% CI [−0.00117, −0.00026]); 11 of 13 participants favor SAPF on this endpoint. **Appendix F.2** reports participant-level contrasts and pointwise bootstrap intervals. Together, these analyses support participant-composition robustness within the auditory cohort and provide additional task-transfer evidence through two Brain refit directions on the same 12 held-out participants.
AuthorsOpenReview ↗

Response to Reviewer F1v5 (Part 4/4: Q3)

Q3: Since the verifier-on and verifier-off conditions yield identical reported F1 scores, what measurable benefit does verification provide? Can the authors demonstrate that the complete forecasting-and-intervention pipeline improves task accuracy or learning outcomes in a sufficiently powered evaluation?

A8: The verifier eliminates action–type incompatibilities in the direct-generation configuration: 44 incompatible candidates are removed and the retained violation rate falls from 11.37% to 0%. We measured this by toggling acceptance on the same 387 natural candidates, with model outputs, evidence, questions, order, and initial budget fixed. In the optimized configuration, compatible-action templates already satisfy the checked rules, explaining the identical on/off results. The seven-participant pilot reports an accuracy point estimate of +5.7 percentage points; this exploratory comparison does not establish an accuracy improvement, while confidence calibration, helpfulness, and trust favor item-specific assistance.

R3-Table6: Verification of the same naturally generated candidates

ConditionRetained candidatesIncompatible-action rateSpan F1Item coverage
Verifier off38711.37%0.449093.00%
Verifier on3430%0.416988.33%
Count-matched random filter, mean34311.34%0.402586.10%

The verifier's operational contribution is enforcing action–type compatibility; Table 6 shows this alongside candidate coverage and localization. In the optimized FOCUS evaluation, independent span F1 improves by 0.0748 over public vote (95% CI [0.0134, 0.1360]); Appendix L.2–L.3 report the two evaluations.

The pilot evaluates assistance content on a separate reasoning stream; its accuracy, calibration, helpfulness, and trust outcomes are summarized in Table 7.

R3-Table7: Pilot outcomes, item-specific minus generic condition (n = 7)

OutcomePaired difference95% participant-bootstrap CI
Confidence Brier−0.0332[−0.0648, −0.0039]
Accuracy (percentage points)+5.7[−11.4, +18.6]
Helpfulness+1.557[1.200, 1.943]
Trust+1.157[0.757, 1.571]

Forecasting, online allocation, localization, and assistance content are evaluated on their corresponding task streams and endpoints, as described in Sections 3.4 and 5.6 and Appendix L.

Manuscript locations

Revisions are highlighted in blue at the manuscript locations below.

  • Appendix F.2, lines 1656–1745 (pp. 31–32) reports stateful catalogue perturbations, participant-level contrasts, and leave-one-out/subset analyses.
  • Appendix F.1, lines 1586–1655 (pp. 30–31) reports complementary cross-task source refits; Appendix F.4, lines 1795–1834 (p. 34; Tables S29–S30) reports the frozen historical-compatibility gate and initial-weight sensitivity.
  • Section 5.6, lines 443–463 (p. 9) reports online allocation and assistance outcomes.
  • Appendix L.2, lines 2133–2151 (p. 39) reports type-matched localization. Appendix L.3, lines 2152–2176 (p. 40) reports the matched verifier analysis.
  • Appendix L.4, lines 2177–2196 (p. 41) reports pilot estimates and audits two frozen-bank examples; the full bank follows in Appendix L.4 (pp. 41–44).

Participant sensitivity, historical compatibility, and stateful perturbations specify when support-aware routing retains its early benefit. The allocation, localization, and assistance studies report distinct outcomes on task-matched endpoints. Thank you for the thoughtful review; it helped us clarify the forecasting contribution and the evidence for its role in the broader assistance workflow.

Reviewer_EwUKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper works on performance prediction in cognitive psychology experiments, such as n-back, a participant answers many short trials in a row. The paper asks how to predict, before each response, whether the participant will answer correctly. It focuses on new participants, whose task may or may not appear in the training data. The authors propose a method, SAPF, first checks whether the current experiment ID appears in the source training data. If it does, SAPF uses the source model, an ensemble of logistic regression and gradient-boosted trees. Otherwise, SAPF predicts from the participant’s own past answers by combining several Beta estimators with exponential weighting. The second method, SafeMix, ignores the ID and weights the source model by its past errors. The experiments use five public datasets: one to train the source model, three for development, and one auditory-oddball dataset for testing. All predictions are made before each response and scored with Brier, NLL, AUROC, and related metrics. On the test set, SAPF reaches NLL 0.066 and AUROC 0.929. When IDs are misleading, SafeMix performs better than SAPF. The paper finds that the ID check works when IDs reliably show whether the source model applies, while learning from past errors works when they do not.

Critical strengths
  1. The paper is well organized, and the figure and tables are clear and complete. It is easy to follow and read.

  2. The paper clearly separates predicting before a response from estimating after it. And the statistical analysis is careful.

Critical weaknesses
  1. Limited contribution to ML community. The research problem is quite interesting, however, I am not sure the new methods offer enough new contribution to ML community. The fallback combines Beta-Binomial counts with exponential weighting. The source model is a standard logistic regression and gradient-boosted tree ensemble. The problem setting and topic may be a better fit for venues on user modeling or educational data mining, or methods in cognitive psychology etc.

  2. The agent part is disconnected from the forecaster and is underspecified. The forecaster works on n-back and auditory-oddball trials, while the agent works on text reasoning items. These tasks do not match, so the forecaster never actually dispatches the agent. The paper does not state which models or prompts the agent uses. Its type F1 is lower than the public-vote baseline, and the verifier does not change any result. In the pilot, some agent questions only restate the claim, while others nearly give the answer.

Questions for authors

See weakness

9 comments
AuthorsOpenReview ↗

Response to Reviewer EwUK (Part 1/3: W1)

Thank you for raising the learning value of source eligibility and its relation to assistance. We added matched source-retention, transfer, and verifier analyses to clarify what each component contributes.

W1: Limited contribution to ML community. The research problem is quite interesting, however, I am not sure the new methods offer enough new contribution to ML community. The fallback combines Beta-Binomial counts with exponential weighting. The source model is a standard logistic regression and gradient-boosted tree ensemble. The problem setting and topic may be a better fit for venues on user modeling or educational data mining, or methods in cognitive psychology etc.

A1: SAPF makes a prequential source-eligibility decision using a fixed training catalogue. Equation (9) separates eligibility disagreement from fallback aggregation regret; matched EWA holds the source and fallback forecasts fixed to measure the cold-start effect.

SAPF selects eligibility, commits a prediction, then scores the response and updates later states. Counts update after each response; fallback weights update on fallback trials.

R2-Table1: Information governing source participation in the evaluated methods

MethodSource participationPost-response update
SAPFFixed source-training catalogueParticipant/context counts; fallback weights on fallback trials
Source + fallback EWAPast losses determine source weightTwo expert weights
SafeMixPast losses determine source weightFive expert weights and online states
Two-expert fixed-sharePast losses and weight renewalExpert weights with a fixed share step

All methods predict before the current response; SAPF uses training support to decide whether source knowledge enters before target feedback establishes its reliability. The matched comparison tests this decision with forecasts fixed.

Equation (9) connects the decision to cumulative squared loss. DT(Q)D_T(Q) sums positive gaps on catalogue disagreements; the second term bounds aggregation regret on fallback trials.

LTSAPF−LTQ≤DT(Q)+PUlog⁡Kη,0<η≤12.L_T^{\mathrm{SAPF}}-L_T^Q\leq D_T(Q)+\frac{P_U\log K}{\eta},\qquad 0<\eta\leq\tfrac12.

Here PUP_U counts participants with fallback trials; KK is the shrinkage grid size. Comparator QQ uses the best fixed shrinkage expert on SAPF's fallback trials with identical candidate forecasts.

Matched source-retention comparison. To isolate the eligibility decision from expert composition, matched EWA receives exactly the same source predictor and complete SAPF fallback forecasts. It starts from equal source/fallback weights and updates them at η=0.25\eta=0.25 or 0.50.5; participants, trial order, and feedback are fixed.

R2-Table2: Matched source-retention comparison with fixed forecasts (n = 13)

MethodBrier (trial 1)Brier (first 5)Brier (first 20)Full BrierFull NLL
Source only0.52780.16680.07090.08430.3353
Matched EWA (η = 0.25)0.36870.12700.05760.01790.0716
Matched EWA (η = 0.50)0.36870.12680.05740.01770.0696
SAPF (catalogue route)0.36870.09910.05010.01740.0671

With source and fallback forecasts fixed, SAPF lowers first-five Brier by approximately 22% relative to matched EWA at both learning rates; full-stream NLL is also lower for SAPF in both matched comparisons. At η = 0.25 and 0.50, the paired differences are −0.0280 (95% CI [−0.0388, −0.0116]) and −0.0277 (95% CI [−0.0385, −0.0115]), respectively. Varying the matched EWA source prior from 0.10 to 0.90 leaves SAPF lower on first-five Brier in all ten settings. The comparison isolates the cold-start source-eligibility effect across initial weights.

Six Brain refits test source utility in two training directions on the original non-training split, separate from the auditory cohort. Appendix F reports the refits and perturbations.

AuthorsOpenReview ↗

Response to Reviewer EwUK (Part 2/3: W2a–W2c)

W2a: The agent part is disconnected from the forecaster and is underspecified. The forecaster works on n-back and auditory-oddball trials, while the agent works on text reasoning items. These tasks do not match, so the forecaster never actually dispatches the agent.

A2: We evaluate SAPF forecasting and online allocation on cognitive-task streams and FOCUS localization and assistance content on textual reasoning streams, with endpoints matched to each task. This separation keeps the pre-response eligibility analysis aligned with its forecasting task while evaluating localization and assistance on their own task stream. Sections 2, 3.4, and 5.6 and Appendix L give the full mapping.

W2b: The paper does not state which models or prompts the agent uses.

A3: Appendix L.1 now specifies the model, prompts, selection, and rendering for both configurations. The saved API model identifier is deepseek-v4-flash; both use temperature 0 and JSON output. The optimized FOCUS configuration disables thinking and combines two candidate generators before selecting at most two spans.

R2-Table3: Agent configurations and generation budgets

SettingFOCUS localizationAssistance pilot
InputArgument, ten type definitions, five curated examplesFacts, rules, and claim; no gold answer
Generation and selectionAdversarial critique and five-shot candidates; agreement-first selection, then router/baseline candidates; at most two spansOne generated question per item
Output budget3,000 tokens per call120 tokens
RenderingDeterministic action-compatible templatesFrozen Chinese translation of the generated question
Configuration35 development-training items; prompt exemplars excluded20-item bank frozen before participant responses

FOCUS retrieves the shortest exact contiguous substring supporting the predicted type, deduplicates candidates, and renders selected spans with action-compatible templates; public vote is the comparator. The pilot generates one question from each item's facts, rules, and claim, applies form and literal-string checks, and freezes the displayed bank before responses. Appendix L.1 contains the full prompts.

W2c: Its type F1 is lower than the public-vote baseline

A4: The primary gain is localization: independent span F1 rises from 0.438 to 0.513, and the type-matched span F1 is 0.115 for both methods. The public-vote advantage on type macro F1 reflects upstream type assignment, while FOCUS optimizes evidence-span localization after the predicted type is supplied. Type macro F1 uses all 300 items, whereas the span metrics use the same 182 gold-positive items; the endpoints therefore separate localization quality from type prediction.

R2-Table4: Localization and type endpoints on the frozen 300-item test set

MethodIndependent span F1Type-matched span F1Type macro F1
Public vote0.4380.1150.135
Optimized Agent, five runs0.513 ± 0.004 (+0.075)0.115 ± 0.0060.110 ± 0.005

Across five runs, the Agent raises independent span F1 by 0.0748 (95% paired run/item-bootstrap CI [0.0134, 0.1360]); type-matched span F1 remains 0.115 for both methods, isolating the localization gain. Section 5.6 and Appendix L.2 report both endpoints.

AuthorsOpenReview ↗

Response to Reviewer EwUK (Part 3/3: W2d / W2e)

W2d: the verifier does not change any result.

A5: The verifier enforces action–type compatibility. We replayed the same 387 naturally generated candidates with identical inputs and toggled acceptance. In the optimized five-run configuration, deterministic action templates already satisfy the compatibility rules, so verification leaves F1 unchanged.

R2-Table5: Matched verifier ablation on naturally generated candidates

ConditionRetained candidatesIncompatible-action rateSpan F1Type F1Item coverage
Verifier off38711.37%0.44900.088493.00%
Verifier on3430%0.41690.073288.33%
Random filter, mean34311.34%0.40250.084286.10%

The verifier removes all 44 incompatible candidates. The count-matched random filter retains an 11.34% mean incompatibility rate, while Table 5 shows the corresponding coverage and localization scores. The measured benefit is targeted compatibility enforcement.

W2e: In the pilot, some agent questions only restate the claim, while others nearly give the answer.

A6: Two examples illustrate the question content. For “The mouse does not need the cat,” the question “Does the mouse need the cat according to the given rules?” restates the relation being judged. The rabbit-visiting question states both the rule “If someone sees the squirrel then they visit the rabbit” and the premise that the rabbit sees the squirrel. This is a near-answer failure mode in which the question supplies the inference needed to judge the claim.

Appendix L.4 includes both items and the full frozen bank. Generation receives facts, rules, and the claim, but not the gold answer; automated checks enforce question form, length, and literal answer-string exclusion. The examples show how the question's reasoning content complements these checks. The pilot measures responses to the frozen content:

R2-Table6: Pilot outcomes, item-specific minus generic condition (n = 7)

OutcomePaired difference95% participant-bootstrap CI
Confidence Brier−0.0332[−0.0648, −0.0039]
Accuracy (percentage points)+5.7[−11.4, +18.6]
Helpfulness+1.557[1.200, 1.943]
Trust+1.157[0.757, 1.571]

The pilot estimates an accuracy difference of +5.7 percentage points. Item-specific assistance lowers confidence Brier and receives higher helpfulness and trust ratings from all seven participants. These outcomes reflect the complete item-specific condition, including its evidence and prompt specificity; the verifier replay separately tests candidate acceptance.

Manuscript locations

Revisions are highlighted in blue at the manuscript locations below.

  • Section 3.5, lines 216–247 (p. 5) presents the loss decomposition. Equation (9), line 240 separates catalogue-disagreement cost from fallback aggregation regret.
  • Section 5.5, lines 424–442 (pp. 8–9) and Appendix F.3, lines 1746–1794 (pp. 33–34) report matched source retention and the EWA protocol. Appendix F.1, lines 1586–1655 (pp. 30–31) reports source refits; Appendix F.4, lines 1795–1834 (p. 34; Tables S29–S30) reports held-out historical eligibility decisions and the ten matched-EWA prior/rate settings.
  • Section 5.6, lines 443–463 (p. 9) reports online allocation and Agent/pilot outcomes.
  • Appendix L.1, lines 2061–2132 (pp. 38–39) specifies Agent models, prompts, generation budgets, and rendering.
  • Appendix L.2, lines 2133–2151 (p. 39) reports independent and type-matched localization.
  • Appendix L.3, lines 2152–2176 (p. 40) reports the matched verifier comparison and retained-candidate, coverage, and F1 results.
  • Appendix L.4, lines 2177–2196 (p. 41) reports pilot estimates and audits two frozen-bank examples; the full bank follows in Appendix L.4 (pp. 41–44).

With identical forecasts, SAPF improves cold-start Brier over matched EWA at both learning rates and across all tested source priors. Together with Equation (9), this evidence isolates the value of making a source-eligibility decision before target feedback accumulates. Thank you for the detailed questions; they helped us sharpen the learning contribution and connect each claim to its evaluation.

Reviewer_riTPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SAPF, Support-Aware Prequential Forecaster, for advance forecasting of cognitive-task performance under distribution shifts. The method uses a fixed catalogue of identifiers to select to whether retain source predictions or switch to hierarchical Beta adaptation for unseen tasks. The paper also introduce a feedback-weighted baseline SafeMix for misleading catalogue membership. Experiments cover represented and shifted cohorts, demonstrating the effectiveness of the proposed method.

Critical strengths
  1. The paper is well written and easy to follow.

  2. The proposed method is straightforward and reasonable.

  3. The analysis of source retention and adaptation is useful.

Critical weaknesses
  1. The main components of the method, including prequential prediction, hierarchical shrinkage, exponential weighting, and fixed-share updates, are techniques that already proposed or widely studied. The main additional mechanism proposed by the method is only the experiment-ID routing, which limits the contribution of the paper.

  2. The main contribution, experiment-ID routing, is insufficiently validated. All tasks in the external experiments are absent from the source catalogue, so SAPF always uses its fallback predictor. However, to demonstrate the ability of the routing, it is important to provide evidences that experiment-ID membership is a reliable criterion for deciding when to retain or discard the source model.

  3. The improvement of the proposed method is limited compared with the strong baseline SafeMix in Table 2.

  4. The conclusion of the proposed method is inconsistent. Immediate fallback benefits SAPF under unsupported cold starts, feedback weighting benefits SafeMix under identifier conflict, and recovery is primarily demonstrated by the standard fixed-share comparator. These findings are informative, but it is hard to find a unified benefits or choosing among these strategies before deployment.

Questions for authors

Please refer to the weaknesses.

9 comments
AuthorsOpenReview ↗

Response to Reviewer riTP (Part 1/3: W1)

Thank you for focusing on when source knowledge should enter a target prediction. We added matched source-retention, bidirectional refit, and stateful routing analyses to test how training support and target feedback govern that decision.

W1: The main components of the method, including prequential prediction, hierarchical shrinkage, exponential weighting, and fixed-share updates, are techniques that already proposed or widely studied. The main additional mechanism proposed by the method is only the experiment-ID routing, which limits the contribution of the paper.

A1: SAPF's contribution is a support-aware prediction rule that acts before target feedback is available, together with an analysis of the separate costs of routing and online adaptation. For catalogue membership S(ct)S(c_t), the committed prediction is

p^t=S(ct)ptsrc+[1−S(ct)]∑k=1Kwi,k,t−1pt,k.\widehat p_t=S(c_t)p_t^{\mathrm{src}}+[1-S(c_t)]\sum_{k=1}^{K}w_{i,k,t-1}p_{t,k}.

The catalogue is fixed from source training; the fallback probabilities and weights use history through t−1t-1. This gives SAPF an immediate source-exclusion mechanism at an unsupported task boundary. Training exposure supplies an initial eligibility signal; observed target losses subsequently guide adaptation within the fallback branch. The current response is scored before any subsequent update.

Equation (9) explains why this decision matters independently of the choice of aggregator:

LTSAPF−LTQ≤DT(Q)+PUlog⁡Kη,0<η≤12.L_T^{\mathrm{SAPF}}-L_T^Q\leq D_T(Q)+\frac{P_U\log K}{\eta},\qquad 0<\eta\leq\tfrac12.

Here DT(Q)D_T(Q) is the positive loss cost of catalogue disagreement with comparator QQ, PUP_U counts participants with fallback trials, and KK is the number of shrinkage experts. The comparator uses the source on its supported route and each participant's best fixed shrinkage expert on SAPF's realized fallback set; the same candidate predictions enter the comparison. The bound separates the cost of source selection from the aggregation regret within the fallback branch.

Our added experiments examine source retention, expert composition, and eligibility-dependent online updates. A matched EWA now uses the same source and complete SAPF fallback forecasts: first-five external Brier is 0.1270 at η=0.25\eta=0.25 and 0.1268 at η=0.5\eta=0.5, versus 0.0991 for SAPF. This isolates the benefit of immediate exclusion from differences in fallback composition. The refits and stateful perturbations identify the complementary roles of source compatibility and subsequent feedback. Appendix F includes the matched comparison alongside the refits and stateful analysis.

AuthorsOpenReview ↗

Response to Reviewer riTP (Part 2/3: W2 & W3)

W2: The main contribution, experiment-ID routing, is insufficiently validated. All tasks in the external experiments are absent from the source catalogue, so SAPF always uses its fallback predictor. However, to demonstrate the ability of the routing, it is important to provide evidences that experiment-ID membership is a reliable criterion for deciding when to retain or discard the source model.

A2: The external auditory cohort tests the cold-start value of excluding an unsupported source: SAPF lowers first-five Brier from 0.1385 to 0.0991 relative to SafeMix (W3). The bidirectional Brain refits test whether catalogue status tracks source utility across represented and unseen tasks.

We fitted six source models across two training directions and three seeds on the Brain dataset. Each direction uses its original training participants and unchanged hyperparameters, then evaluates the original non-training split. The external auditory evaluation uses a separate cohort. The refits place represented and unseen tasks in the same evaluation design. Scores below are participant-balanced Brier, with seeds averaged within participant.

R1-Table1: Cross-task source refits by catalogue status

Source trainingEvaluation statusParticipantsSource onlyFallbackLower Brier
Experiment 1Represented50.14960.1548Source
Experiment 1Unseen70.11170.1291Source
Experiment 2Represented70.16140.1291Fallback
Experiment 2Unseen50.21120.1548Fallback

Define the exclusion cost as

Δ_excl = Brier_fallback − Brier_source

On unseen tasks, it is +0.0175 in one direction (95% participant-bootstrap CI [0.0046, 0.0317]) and −0.0564 in the reverse direction (CI [−0.1338, −0.0064]), identifying when the source should be retained or excluded. Catalogue membership records training exposure; source–task compatibility determines the exclusion benefit.

A historical-compatibility gate, fixed from leave-one-participant-out forecasts on the eight source-training participants, selects the lower-loss branch in all four held-out task-direction comparisons. It agrees with held-out outcomes in 19/24 participant-direction cells, versus 11/24 for exact catalogue membership. Historical compatibility therefore provides a complementary pre-feedback eligibility signal; Appendix F reports the refits, validation, and stateful sensitivity.

W3: The improvement of the proposed method is limited compared with the strong baseline SafeMix in Table 2.

A3: SAPF's clearest advantage over SafeMix appears at cold start: first-five Brier is 28.5% lower and remains 21.0% lower over the first 20 trials. Across the full stream, SAPF lowers NLL by 7.4%. R1-Table 2 quantifies these three gains.

R1-Table2: Cold-start Brier and full-stream NLL versus SafeMix (n = 13)

Metric and windowSAPFSafeMixSAPF − SafeMix (95% paired CI)Relative reduction vs SafeMix
Brier, first 50.09910.1385−0.0395 [−0.0540, −0.0177]28.5%
Brier, first 200.05010.0634−0.0133 [−0.0189, −0.0071]21.0%
NLL, full stream0.06710.0725−0.0054 [−0.0082, −0.0022]7.4%

Relative reduction uses participant means; positive values indicate lower SAPF loss. These gains follow the routing mechanism: SAPF excludes the unsupported source immediately, whereas SafeMix learns to discount it from feedback. The same ordering persists across leave-one-participant-out estimates and overlapping six-person subsets through trial 20. Matched EWA in W1 isolates immediate exclusion with both forecasts fixed. Appendix F.2 reports participant-level stability; intervals are pointwise participant-bootstrap estimates.

AuthorsOpenReview ↗

Response to Reviewer riTP (Part 3/3: W4)

W4: The conclusion of the proposed method is inconsistent. Immediate fallback benefits SAPF under unsupported cold starts, feedback weighting benefits SafeMix under identifier conflict, and recovery is primarily demonstrated by the standard fixed-share comparator. These findings are informative, but it is hard to find a unified benefits or choosing among these strategies before deployment.

A4: The experiments address three decisions in source transfer: eligibility before feedback, source weighting after feedback, and weight recovery after a reliability change. SAPF fixes eligibility from identifiers observed during source training and applies that rule before each response. SafeMix and two-expert EWA keep the source available and learn its weight from past losses. Fixed-share adds weight renewal for changing reliability. These mechanisms explain the different outcomes across cold starts, metadata mismatch, and source restoration.

R1-Table3: Source-transfer decisions and their experimental evidence

DecisionMethod responsibleKey evidence
Pre-feedback eligibilitySAPF catalogueFirst-five Brier: 0.0991 vs 0.1385 (SafeMix)
Feedback adaptationSafeMix / EWASource utility changes across Brain refit directions
Reliability recoveryFixed-share renewalRestored-phase Brier: 0.1290 vs 0.1446 (EWA) and 0.1453 (SafeMix)

Across these settings, SAPF fixes pre-feedback eligibility from source-training support and updates fallback experts after each response; SafeMix and EWA adapt source weight from losses, while fixed-share renews that weight when reliability returns. The historical gate provides an independent pre-feedback eligibility check, and controlled restoration yields Brier 0.1290 versus 0.1446 for two-expert EWA and 0.1453 for SafeMix, separating the three roles.

Manuscript locations

Revisions are highlighted in blue at the manuscript locations below.

  • Section 3.5, lines 216–247 (p. 5) presents the routing-loss decomposition and aggregation guarantee. Equation (9), line 240 states the combined bound.
  • Section 5.5, lines 424–442 (pp. 8–9) reports matched source retention and the controlled recovery comparison.
  • Appendix F.1, lines 1586–1655 (pp. 30–31) reports bidirectional source refits and natural-boundary analysis.
  • Appendix F.2, lines 1656–1745 (pp. 31–32) reports stateful catalogue perturbations and participant sensitivity.
  • Appendix F.3, lines 1746–1794 (pp. 33–34) specifies the matched EWA protocol and results.
  • Appendix F.4, lines 1795–1834 (p. 34) reports historical eligibility validation and matched-EWA initial-weight sensitivity.

These results identify SAPF's contribution as a pre-feedback eligibility decision: it improves cold-start prediction when source-training support is informative, while feedback weighting and fixed-share address changing transfer utility. Thank you for the careful review; it helped us make these complementary roles explicit.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers for their thoughtful feedback and continued discussion. We appreciate their recognition of the pre-response prediction formulation, participant-disjoint evaluation, and source-retention analysis.

We have added matched comparisons, controlled source refits, and component analyses to address the reviewers' questions. Below, we summarize our core contributions, experimental updates, and clarifications in the revised manuscript and follow-up responses.

For clarity, we refer to Reviewers riTP, EwUK, and F1v5 as R1, R2, and R3, respectively.


Core Contributions of Our Work

  1. Pre-Response Forecasting Formulation: We formulate cognitive-performance forecasting using information available before the current response, with testable temporal invariance and a regret decomposition separating source-route disagreement from fallback adaptation.
  2. Support-Aware Transfer and Online Adaptation: SAPF combines catalogue-based source selection with hierarchical Beta adaptation for unseen tasks. The complementary SafeMix rule learns source influence from past predictive losses when transferability must be inferred from feedback.
  3. Evaluation under Participant and Domain Shift: We evaluate source retention, suppression, and recovery through participant-disjoint comparisons, two-expert ablations, and temporal perturbations. Prefix-causal allocation further tests how pre-response forecasts target likely errors under an online assistance budget.

Updates of Experimental Results During Rebuttal

  • Section 5.5 and Appendix F.3: Added matched EWA using identical source and fallback forecasts; first-five Brier is 0.099 for SAPF versus 0.127 for matched EWA (R1/R2).
  • Appendix F.1–F.4: Added bidirectional source refits, historical compatibility, catalogue perturbations, and participant sensitivity. Early Brier and full-stream NLL gains over SafeMix persist across all 13 leave-one-participant-out estimates and 1,716 overlapping six-person subsets (R1–R3).
  • Section 5.6 and Appendix L: Reported budgeted allocation, type-matched localization, matched verifier analysis, and assistance-content outcomes. SAPF covers 42.16% of auditory errors at a 5.85% call rate (R2/R3).
  • R1/R2 follow-up responses: Added a controlled ID-load study with 120 source fits and 62 participants. Joint support improves Brier and NLL over five prior-assignment controls under identical updates. The condition-aware extension achieves Brier/NLL 0.152/0.458, versus 0.153/0.461 for matched EWA at η = 0.50.
  • R2 follow-up response: Added forecast-triggered Agent execution, content-preserving repair, and type-aware fusion. Repair corrects 44 incompatibilities while retaining all 387 candidates. Fusion raises type macro F1 from 0.135 to 0.147 and type-matched span F1 from 0.115 to 0.140 in the added evaluation.

Updates of In-Depth Discussions During Rebuttal

  • Section 3.5: Clarified the separate roles of source-selection cost and fallback aggregation regret (R1/R2).
  • Sections 2, 3.4, and 5.6: Sharpened the forecasting contribution and distinguished probability estimation, error allocation, localization, and assistance outcomes (R2/R3).
  • Appendix L.1–L.4: Specified Agent models and prompts, verification rules, evaluation endpoints, and pilot procedures (R2/R3).
  • R1/R2 follow-up responses: Explained how joint experiment-condition coverage informs source participation and distinguished the condition-aware extension from original exact-ID SAPF.

Revisions in the revised manuscript and appendix are highlighted in blue. The latest experiments are detailed in the R1/R2 follow-up responses.

We sincerely thank the reviewers and the Area Chair for their time, thoughtful feedback, and consideration of our revised manuscript and additional results. We welcome further questions and discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.