- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper investigates the effectiveness of test-time scaling and post-training for individual stance prediction: given a user’s discussion history, a target discussion, and its candidate stances, the task is to predict the stance the user expresses conditional on participation. The authors introduce STANCE-BENCH, comprising 2,499 prediction tasks from 500 Hacker News users, and separately record candidate coverage and final selection. Across several inference-time strategies and learning methods, the paper reports four phenomena: incorrect consensus, selection failure, response overfitting, and an early plateau in reinforcement learning. It further explores a method that combines direct label scores with comment-level assessments of historical evidence. On the 781-task test set, this method achieves a discussion-specific Macro F1 of 21.83, compared with 19.27 for direct scoring, although the primary paired comparisons do not pass Holm correction. The main contribution is a diagnostic analysis of individual stance prediction, supported by a benchmark and an exploratory method.
Critical strengths
1.The paper separately evaluates candidate generation and final selection in individual stance prediction, providing a more informative analysis than final accuracy alone. Figure 3 shows that some methods are primarily limited by insufficient candidate coverage, whereas others improve coverage but fail to select the target user’s stance effectively. This decomposition helps explain distinct sources of failure behind similar aggregate performance and provides concrete directions for subsequent method development. 2.The experiments cover several inference-time strategies across three models, as well as verification, supervised fine-tuning, reinforcement learning, user profiles, and activation steering on Qwen3-8B. Several experiments report results over three seeds, and the paper provides matched controls for some comparisons. The evidence-fusion method includes branch ablations, history replacement, and prompt-concatenation controls. The authors also explicitly disclose that the primary comparisons do not pass Holm correction. These choices strengthen the study’s diagnostic value and help readers interpret the exploratory findings appropriately.
Critical weaknesses
1.The central findings depend on the reference labels and evaluation protocol of STANCE-BENCH, yet the current manuscript does not provide sufficient validation information. Section 3 describes model-assisted annotation and the use of target-user contributions to audit and revise stance taxonomies, but it does not report the size of the human audit, agreement measurements, or correction outcomes. Page 10 mentions volunteer annotators without supplying this supporting evidence.The main text and appendix also do not fully specify the training, validation, and test partitions; user and discussion overlap across partitions; the construction and isolation of additional training tasks for the profile methods; or the filtering statistics for OTHER examples. With only 781 test tasks distributed across 497 reference-supported discussion–stance classes, the resampling unit and treatment of within-user and within-discussion dependence are particularly important, but remain unspecified. These omissions limit the assessment of label reliability, generalization, and small performance differences. They do not establish that leakage occurred, but prevent readers from adequately ruling out such concerns. 2.Most test-time methods are evaluated only with four candidates, without an analysis across candidate budgets. The PlanSearch adaptation is particularly revealing: diverse generation followed by label voting results in selection of the first candidate on 774 of 781 Flash tasks (§4.2). This clearly demonstrates a limitation of that selection rule, but does not establish that a more suitable selector or a larger budget would also fail to improve prediction.For SFT, the training objective covers the comment-and-label response sequence, whereas evaluation uses only the label. The paper also reports more OTHER predictions, invalid outputs, and generation truncation. Although a matched base-model control is provided, the absence of an error breakdown and a direct label-supervision control makes it difficult to determine how much of the degradation reflects stance discrimination itself. For GRPO, sparse reward contrast co-occurs with a plateau within the 300-step training horizon, but a causal relationship is not established. Thus, the results primarily support descriptive findings for the evaluated configurations; broader interpretations of the failure mechanisms require stronger controls or more explicit qualification.
Questions for authors
1.Please provide the human-audit sampling procedure, sample size, number of annotators, agreement measurements, and correction outcomes. How are user contributions containing multiple comments or positions reduced to a single dominant label? How many discussion taxonomies were revised after inspecting target-user contributions, and do these revisions affect the reported diagnostic findings? This information would directly affect my assessment of benchmark validity. 2.Given that PlanSearch almost always selects the first candidate on Flash, do the main findings persist when the candidate pool is held fixed and selection is performed by a history-conditioned selector? Are results available for candidate coverage and final accuracy across different candidate budgets, to distinguish limitations of the fixed four-candidate setting from broader limitations of test-time scaling? 3.How much of the SFT degradation is attributable to incorrect listed stances, OTHER predictions, formatting errors, and truncation, respectively? Does the degradation persist under consistent output constraints or direct stance-label supervision? For GRPO, does the available evidence support only co-occurrence between sparse reward contrast and the plateau, or are there controls that distinguish the effects of reward structure, training budget, and optimization configuration?
3 comments
Kind Reminder and Summary of Our Response
Dear Reviewer 3FTa,
Thank you again for your instructive review. As the discussion period is closing soon, we would like to kindly check whether our responses have addressed your concerns. We summarize them briefly below.
- W1, Q1. Five annotators collectively labeled all 881 test tasks. Agreement with the reference labels was 93.3% (Brennan-Prediger S = 0.92), and inter-annotator agreement was 89.3% on a common 337-task subset. Exactly one stance applies in 92.8% of annotations, and taxonomy revisions affected 66 of 1,100 discussions (6.0%). Under a discussion-level bootstrap, our method keeps a +2.83 Macro F1 gain over direct scores only.
- W2, Q2. On fixed Qwen3-8B PlanSearch pools at k = 4/8/16, coverage is 72.0/96.8/98.6%. Voting and two history-reading selectors all stay below direct generation (26.9%).
- Q3. Of comment-and-label SFT errors, 88.4% are wrong listed stances, and label-only SFT reaches 20.5% accuracy versus 26.0% for the base. For GRPO, the evidence supports co-occurrence only, and we will qualify the claim for the evaluated configurations.
We would be glad to clarify any remaining point, and would sincerely appreciate it if you could consider updating your assessment.
Best regards,
The Authors
A gentle follow-up before the discussion closes
Dear Reviewer 3FTa,
We hope this message finds you well. We understand this is a busy time, and we just wanted to gently follow up on our response.
Any thoughts you could share before the discussion closes would be greatly appreciated. Thank you again for your time and helpful feedback.
Best regards,
The Authors
Thank you, the key question is how robust the findings are, and here we report the human audit, bootstrap, and new controls.
W1/Q1 · Benchmark validity. Five annotators collectively labeled all 881 test tasks with a listed stance or OTHER, using the same inputs as the LLM labeler. Agreement with reference labels was 93.3% (95% CI [91.5, 94.8]), compared with 12.8% expected by chance (Brennan–Prediger S = 0.92). All five labeled a common subset of 337 tasks, with 89.3% inter-annotator agreement. They resolved disagreements by discussion.
The labeler reads all of a user’s comments in a discussion and assigns the dominant position the user argues for. A position mentioned only to reject it is excluded. A second applicable stance is stored separately. In 92.8% of annotations, exactly one stance applies.
A discussion’s taxonomy was revised once if more than 15% of its contributions, including target-user contributions, were labeled OTHER. Revisions could add stances and relabel OTHER contributions. Before the human audit and model experiments, 66 of 1,100 discussions (6.0%) gained 81 stances and 295 of 8,980 annotations (3.3%) changed.
We split users into 200 training, 50 validation, and 250 test users. Each task follows its user’s split. Of the 781 listed-stance test tasks, 91.8% share a discussion with a development task. Test reference labels are held out from training, and visible target context excludes benchmark users’ comments. The profile methods use 694 extra training and 154 extra validation tasks from earlier labeled discussions of users in the corresponding splits. History ends before the user’s first comment in each new target, and these tasks do not duplicate benchmark tasks. Of 1,249 test tasks, 781 have a listed stance, 100 have OTHER, and 368 have NO_STANCE. The main metrics use the 781 tasks, and secondary Macro F1 uses all 881 listed-stance or OTHER tasks. Training retains OTHER and excludes NO_STANCE. On the 781-task subset, OTHER predictions count as errors.
Excluding the 123 tasks in revised discussions or the 52 tasks with visible comments posted after the user’s first target comment preserves the direction of all four Macro F1 comparisons below. Values are point differences.
The original bootstrap resamples users, keeping each user’s tasks together. We also ran 2,000 bootstrap replicates resampling the 245 test discussions, keeping each discussion’s tasks together. Below are the resulting 95% intervals for paired differences, in points.
We will report both resampling analyses in the revision.
W2/Q2 · Selection and budgets. On fixed Qwen3-8B PlanSearch pools, we compared voting with two selectors that read the user’s history. All values are percentages.
All selectors scored below direct generation (26.9%). At k = 4, likelihood accuracy was 25.1% without history and 23.9% with another user’s history. The reviewer stayed near 24%. With independent samples at k = 1/2/4/8/16, coverage was 28.0/37.5/45.8/53.8/61.6%, while majority and likelihood accuracy stayed within 26.4–28.6%.
Q3 · SFT and GRPO. Of comment-and-label SFT errors, 88.4% were wrong listed stances, 9.6% OTHER, 0.8% truncations, and 1.1% other invalid outputs. Using 713 training tasks, 270 updates, and three seeds per objective, generation accuracy was 20.4% for comment and label, 20.9% with 30× label weighting, and 20.5% for label only, versus 26.0% for the base. Label-only SFT produced no invalid or truncated outputs. Direct scoring of listed labels gave 25.5%, 26.6%, and 27.3%, versus 25.9% for the base.
We do not claim that sparse reward contrast caused the GRPO plateau. In the same three 300-step runs, most validation gains occurred by step 150 and reward contrast stayed sparse. In exploratory runs with NO_STANCE as a candidate, the exact-match with KL, prior-corrected, coverage-dual, Brier, and Dr. GRPO variants all scored below their matched bases. These runs did not isolate the effects of reward, optimization, and training budget. We will add the analyses and qualify the claims for the evaluated configurations.