- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes PACT, a framework for training wearable language agents to decide whether to intervene, what assistance to provide, and how to deliver it. Executable decision rules generate supervision and label-changing counterfactual pairs, and provide rewards for reinforcement learning. The method groups rollouts across paired situations to preserve reward contrast when within-input rewards are identical. The paper also introduces POISE, a benchmark of 8,559 decisions across twelve scenarios. Experiments compare prompting and training methods on POISE, with additional evaluation on CAB.
Critical strengths
-
The paper addresses an important problem for proactive agents: deciding when assistance is warranted while respecting delivery and timing constraints. Evaluating both intervention and abstention makes the task relevant to continuously available wearable assistants.
-
Retaining executable decision rules provides a coherent basis for supervision, counterfactual construction, and reward computation. This design makes the labeling process inspectable and enables controlled evaluation of responses to changes in decision-relevant conditions.
-
POISE extends evaluation beyond intervention accuracy to tools, delivery channels, evidence, safety, and timing. The reward-component ablations help reveal capability differences that the aggregate score can obscure, while the leave-one-scenario-out experiments provide useful evidence about transfer limitations.
Critical weaknesses
-
The practical advantage of learning the rules with a language model is not established. PACT derives intervention labels, tool constraints, and delivery requirements from executable specifications, but the experiments do not compare against directly executing these rules or using a language model to extract the state before rule-based decision making. These comparisons are particularly relevant because the decision mechanism is already available in executable form. Without them, it remains unclear whether distilling the rules into a language model offers advantages in accuracy, efficiency, or generalization over a simpler rule-based pipeline.
-
The core counterfactual grouping mechanism lacks an isolated ablation. PACT changes both the reward components and the composition of the groups used to normalize advantages, while the reported ablations primarily examine reward components. I would like to see ordinary within-input GRPO, random cross-input grouping, and counterfactual grouping compared under the same reward function, training data, and rollout budget. This would clarify whether the gains depend on meaningful counterfactual pairing or simply on introducing reward differences across inputs. The current results do not isolate the contribution of the proposed grouping mechanism.
-
The substantial performance gap across backbones is not explained. In Table 4, Qwen2.5-1.5B and Qwen3-1.7B perform similarly under Direct prompting, but their POISE scores diverge to 83.1 versus 42.1 after SFT and 89.0 versus 46.6 after PACT. PACT improves both models, so this gap should not be attributed solely to its reinforcement learning stage; much of it is already present after SFT. However, the paper reports shared hyperparameters without investigating convergence, model configuration, or training compatibility. Targeted analysis would help establish the conditions under which the framework works well and clarify its applicability across backbones.
Questions for authors
Refer to the weaknesses.
2 comments
Authors' Rebuttal (2/2)
Continued: Authors’ Rebuttal (2/2)
[Remaining Part of Our Response to Reviewer W2] Opposite-label random pairs differ in reward across inputs in almost every group, yet they remain 2.4 POISE and 2.8 Pair-C. points below counterfactual pairs. Larger within-input groups add only 0.3 points, so group size does not explain the gain either. Reward differences across inputs alone therefore do not account for the gain, which requires pairing situations that differ only in the decisive variable. We have added Table R3 to App. B.6 of the revised manuscript.
W3 [Performance gap between Qwen2.5-1.5B-Instruct and Qwen3-1.7B]
Thank you for your comments. The shared SFT configuration accounts for most of the gap, and with a backbone-specific configuration, Qwen3-1.7B reaches 82.5 POISE after SFT and 88.1 after PACT, a gain of 5.6 points close to the 5.9 points on Qwen2.5-1.5B-Instruct. Specifically, we address this from two aspects below:
(1) Failure Mode under the Shared Configuration. After SFT under the shared configuration, Qwen3-1.7B intervenes on 79.4% of the situations that warrant silence, consistent with its Acc-P of 56.1 in Tab. 4, against 16.0% for Qwen2.5-1.5B-Instruct (Tab. 22), while both rarely miss a warranted intervention (8.5% vs. 6.7%). PACT still adds 10.3 Pair-C. points on this checkpoint, more than the 8.2 points on Qwen2.5-1.5B-Instruct.
(2) Convergence and Configuration. Under each configuration in Table R4, validation POISE on a held-out 10% of the training split changes by less than one point from the third to the fifth SFT epoch, so the two additional epochs do not close the gap. The shared configuration keeps every SFT setting identical across backbones, including the chat template and the learning rate. Qwen3-1.7B, however, is a hybrid reasoning model with a non-thinking chat template. We have retrained it with this template in both SFT and evaluation and then selected its learning rate on the same held-out split, with all other settings unchanged. All rows use the final SFT checkpoint, and Table R4 below reports the results.
Table R4. SFT and SFT+PACT on POISE under the shared and the backbone-specific SFT configurations.
| Backbone | SFT configuration | Validation POISE, epoch 3 / 5 | SFT POISE | SFT FD ↓ | SFT+PACT POISE | Gain |
|---|---|---|---|---|---|---|
| Qwen2.5-1.5B-Instruct | Shared | 82.4 / 83.0 | 83.1 | 16.0 | 89.0 | +5.9 |
| Qwen3-1.7B | Shared | 41.6 / 42.3 | 42.1 | 79.4 | 46.6 | +4.5 |
| Qwen3-1.7B | Non-thinking chat template | 77.2 / 77.8 | 77.9 | 24.3 | 84.6 | +6.7 |
| Qwen3-1.7B | Non-thinking chat template, learning rate 2×10⁻⁵ | 82.0 / 82.6 | 82.5 | 17.2 | 88.1 | +5.6 |
The non-thinking chat template recovers most of the gap, and the selected learning rate closes most of the remainder. Under this configuration, SFT+PACT also exceeds SFT+Outcome (84.4) and SFT+Judge (86.1) on Qwen3-1.7B.
PACT therefore retains its gain on both backbones with the same reward, grouping, and RL hyperparameters. We have added this analysis and Table R4 to App. B.6, and the backbone-specific SFT settings to App. B.3 of the revised manuscript.
We hope this fully addresses your questions and comments, and remain happy to engage in further discussion for the rest of the rebuttal period.
Authors' Rebuttal (1/2)
Thank you for providing us with your valuable feedback. We have carefully considered your comments and would like to address them as below.
W1 [Learning the rules with a language model versus executing them]
Against rule execution on states that a language model extracts from the rendered context, PACT is more accurate at the same model size, by 7.4 POISE and 12.3 Pair-C. points, and transfers to benchmarks that provide no executable specification. Executing the rules requires the structured state, which the agent receives only as rendered context. On the true state, rule execution reproduces the labels by construction (Sec. 3), so it serves as the oracle reference. A rule-based pipeline must first recover every variable from the description, and misreading the decisive variable flips the decision. Specifically, we address this from two aspects below:
(1) Accuracy and Efficiency. We have added rule-execution baselines in which a language model extracts the state variables of the scenario from the rendered context and the full specification is then executed to produce the decision, tools, and delivery plan. Each extraction model receives the scenario identity and the list of its variables. The 1.5B extraction model is Qwen2.5-1.5B-Instruct fine-tuned on the states of the training split with the SFT configuration of PACT. Table R1 below reports the results.
Table R1. Rule execution on extracted states and the learned policy on the POISE test split. Latency: one A100 GPU for the 1.5B models, official API for DeepSeek-V3.2.
71% of the pairs that the 1.5B pipeline fails contain a misread decisive variable. With the 671B DeepSeek-V3.2 as extractor, the pipeline reaches 88.2 POISE against 89.0 for PACT and a higher Tool-F1, at 7.9 s per sample through its API against 0.38 s for the locally served policy.
(2) Generalization. We have evaluated the POISE-trained models zero-shot on CAB and ProactiveBench (Lu et al., 2025), and Table R2 below reports the results. Unlike the CAB columns of Tab. 4, which come from models trained on the CAB training split, these models never see CAB data.
Table R2. Zero-shot results of Qwen2.5-1.5B-Instruct models trained only on POISE.
PACT transfers best among the evaluated models on both benchmarks, whereas rule execution would first require new specifications for each benchmark.
We have added Tables R1 and R2 to App. B.6 of the revised manuscript.
W2 [Isolated ablation of counterfactual grouping]
Thank you for your comments. Under the same reward, training data, and rollout budget, counterfactual grouping exceeds within-input grouping by 3.4 POISE points and random pairs with opposite labels by 2.4 points. Tab. 4 isolates the reward, because its Outcome, Judge, and PACT variants share counterfactual grouping and differ only in their rewards (Sec. 5). At one rollout per input, where within-input groups carry no advantage by construction, PACT also raises POISE from 83.1 to 84.8 (Fig. 4).
We have added a grouping ablation in which all runs use 16 inputs with five rollouts each per step and train for two epochs. Random cross-input grouping joins two unrelated inputs, and its opposite-label variant joins two inputs of the same scenario with opposite labels. The row with 10 rollouts per input uses 8 inputs per step and thus matches the group size of pairing. Table R3 below reports the results, including the share of zero-variance groups, in which all rollouts receive the same reward.
Table R3. Grouping ablation on Qwen2.5-1.5B-Instruct with the PACT reward, the same SFT initialization and training data, and 80 rollouts per step.
Please see the subsequent Authors' Rebuttal (2/2) for remaining responses.