acceptodds
Under review as a conference paper at ICLR 2027

PACT: Toward Verifiable Proactive Assistance for Wearable Language Agents

Abstract

A wearable agent that continuously observes its user must decide, without an explicit request, whether to intervene, what assistance to provide, and how to deliver it under device and timing constraints. Existing training largely relies on answer-level supervision, which provides target decisions but does not retain an executable mechanism for verifying new policy outputs. Group-relative reinforcement learning poses a second difficulty, as identical rollouts receive identical rewards and yield no advantage signal. We formalize label derivability as the property that the decision mechanism remains executable after data construction, and propose PACT, a training framework that exploits this property for proactive policy learning. PACT derives supervision from executable decision rules over measurable conditions, re-executes these rules to verify policy outputs, and forms optimization groups across label-changing counterfactual pairs to preserve reward contrast when within-input variation disappears. We further introduce POISE, a benchmark of 8559 wearable decisions across twelve scenarios, 49 tools, and nine delivery channels, grounded in sensor and contextual observations. Evaluation on POISE reveals persistent difficulty in distinguishing warranted intervention from abstention among prompting and inference-time methods. PACT improves over supervised fine-tuning and alternative RL objectives on POISE, while also yielding gains in tool selection and proactivity-score calibration on external benchmark.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Proactive wearable agents, executable supervision, verifiable rewards, counterfactual reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_M8mDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PACT, a framework for training wearable language agents to decide whether to intervene, what assistance to provide, and how to deliver it. Executable decision rules generate supervision and label-changing counterfactual pairs, and provide rewards for reinforcement learning. The method groups rollouts across paired situations to preserve reward contrast when within-input rewards are identical. The paper also introduces POISE, a benchmark of 8,559 decisions across twelve scenarios. Experiments compare prompting and training methods on POISE, with additional evaluation on CAB.

Critical strengths
  1. The paper addresses an important problem for proactive agents: deciding when assistance is warranted while respecting delivery and timing constraints. Evaluating both intervention and abstention makes the task relevant to continuously available wearable assistants.

  2. Retaining executable decision rules provides a coherent basis for supervision, counterfactual construction, and reward computation. This design makes the labeling process inspectable and enables controlled evaluation of responses to changes in decision-relevant conditions.

  3. POISE extends evaluation beyond intervention accuracy to tools, delivery channels, evidence, safety, and timing. The reward-component ablations help reveal capability differences that the aggregate score can obscure, while the leave-one-scenario-out experiments provide useful evidence about transfer limitations.

Critical weaknesses
  1. The practical advantage of learning the rules with a language model is not established. PACT derives intervention labels, tool constraints, and delivery requirements from executable specifications, but the experiments do not compare against directly executing these rules or using a language model to extract the state before rule-based decision making. These comparisons are particularly relevant because the decision mechanism is already available in executable form. Without them, it remains unclear whether distilling the rules into a language model offers advantages in accuracy, efficiency, or generalization over a simpler rule-based pipeline.

  2. The core counterfactual grouping mechanism lacks an isolated ablation. PACT changes both the reward components and the composition of the groups used to normalize advantages, while the reported ablations primarily examine reward components. I would like to see ordinary within-input GRPO, random cross-input grouping, and counterfactual grouping compared under the same reward function, training data, and rollout budget. This would clarify whether the gains depend on meaningful counterfactual pairing or simply on introducing reward differences across inputs. The current results do not isolate the contribution of the proposed grouping mechanism.

  3. The substantial performance gap across backbones is not explained. In Table 4, Qwen2.5-1.5B and Qwen3-1.7B perform similarly under Direct prompting, but their POISE scores diverge to 83.1 versus 42.1 after SFT and 89.0 versus 46.6 after PACT. PACT improves both models, so this gap should not be attributed solely to its reinforcement learning stage; much of it is already present after SFT. However, the paper reports shared hyperparameters without investigating convergence, model configuration, or training compatibility. Targeted analysis would help establish the conditions under which the framework works well and clarify its applicability across backbones.

Questions for authors

Refer to the weaknesses.

2 comments
AuthorsOpenReview ↗

Authors' Rebuttal (1/2)

Thank you for providing us with your valuable feedback. We have carefully considered your comments and would like to address them as below.


W1 [Learning the rules with a language model versus executing them]

Against rule execution on states that a language model extracts from the rendered context, PACT is more accurate at the same model size, by 7.4 POISE and 12.3 Pair-C. points, and transfers to benchmarks that provide no executable specification. Executing the rules requires the structured state, which the agent receives only as rendered context. On the true state, rule execution reproduces the labels by construction (Sec. 3), so it serves as the oracle reference. A rule-based pipeline must first recover every variable from the description, and misreading the decisive variable flips the decision. Specifically, we address this from two aspects below:

(1) Accuracy and Efficiency. We have added rule-execution baselines in which a language model extracts the state variables of the scenario from the rendered context and the full specification is then executed to produce the decision, tools, and delivery plan. Each extraction model receives the scenario identity and the list of its variables. The 1.5B extraction model is Qwen2.5-1.5B-Instruct fine-tuned on the states of the training split with the SFT configuration of PACT. Table R1 below reports the results.

Table R1. Rule execution on extracted states and the learned policy on the POISE test split. Latency: one A100 GPU for the 1.5B models, official API for DeepSeek-V3.2.

SystemParametersAcc-PPair-C.Tool-F1Chan.-F1POISESeconds per sample
Qwen2.5-1.5B-Instruct extraction + rules1.5B85.974.682.383.581.60.41
DeepSeek-V3.2 extraction + rules671B91.483.788.988.788.27.9
SFT+PACT (Qwen2.5-1.5B-Instruct)1.5B93.386.986.889.089.00.38

71% of the pairs that the 1.5B pipeline fails contain a misread decisive variable. With the 671B DeepSeek-V3.2 as extractor, the pipeline reaches 88.2 POISE against 89.0 for PACT and a higher Tool-F1, at 7.9 s per sample through its API against 0.38 s for the locally served policy.

(2) Generalization. We have evaluated the POISE-trained models zero-shot on CAB and ProactiveBench (Lu et al., 2025), and Table R2 below reports the results. Unlike the CAB columns of Tab. 4, which come from models trained on the CAB training split, these models never see CAB data.

Table R2. Zero-shot results of Qwen2.5-1.5B-Instruct models trained only on POISE.

ModelCAB IndexCAB ScoreCAB Tool-F1ProactiveBench F1ProactiveBench false-alarm rate ↓
Direct (no training)54.52.345.541.763.5
SFT73.031.858.449.235.1
SFT + GRPO, Outcome74.333.059.150.633.4
SFT + GRPO, Judge73.835.160.350.134.2
SFT + PACT76.538.662.754.426.7

PACT transfers best among the evaluated models on both benchmarks, whereas rule execution would first require new specifications for each benchmark.

We have added Tables R1 and R2 to App. B.6 of the revised manuscript.


W2 [Isolated ablation of counterfactual grouping]

Thank you for your comments. Under the same reward, training data, and rollout budget, counterfactual grouping exceeds within-input grouping by 3.4 POISE points and random pairs with opposite labels by 2.4 points. Tab. 4 isolates the reward, because its Outcome, Judge, and PACT variants share counterfactual grouping and differ only in their rewards (Sec. 5). At one rollout per input, where within-input groups carry no advantage by construction, PACT also raises POISE from 83.1 to 84.8 (Fig. 4).

We have added a grouping ablation in which all runs use 16 inputs with five rollouts each per step and train for two epochs. Random cross-input grouping joins two unrelated inputs, and its opposite-label variant joins two inputs of the same scenario with opposite labels. The row with 10 rollouts per input uses 8 inputs per step and thus matches the group size of pairing. Table R3 below reports the results, including the share of zero-variance groups, in which all rollouts receive the same reward.

Table R3. Grouping ablation on Qwen2.5-1.5B-Instruct with the PACT reward, the same SFT initialization and training data, and 80 rollouts per step.

GroupingPOISEAcc-PFD ↓Pair-C.Tool-F1Chan.-F1Zero-variance groups (%)
None (SFT)83.188.716.078.781.483.7n/a
Within-input (standard GRPO)85.691.013.282.184.085.441.7
Within-input, 10 rollouts per input85.991.212.982.684.285.634.9
Random cross-input85.891.412.483.083.685.19.6
Random cross-input, opposite labels86.692.011.384.184.385.95.8
Counterfactual pairs (PACT)89.093.39.986.986.889.07.1

Please see the subsequent Authors' Rebuttal (2/2) for remaining responses.

AuthorsOpenReview ↗

Authors' Rebuttal (2/2)

Continued: Authors’ Rebuttal (2/2)


[Remaining Part of Our Response to Reviewer W2] Opposite-label random pairs differ in reward across inputs in almost every group, yet they remain 2.4 POISE and 2.8 Pair-C. points below counterfactual pairs. Larger within-input groups add only 0.3 points, so group size does not explain the gain either. Reward differences across inputs alone therefore do not account for the gain, which requires pairing situations that differ only in the decisive variable. We have added Table R3 to App. B.6 of the revised manuscript.


W3 [Performance gap between Qwen2.5-1.5B-Instruct and Qwen3-1.7B]

Thank you for your comments. The shared SFT configuration accounts for most of the gap, and with a backbone-specific configuration, Qwen3-1.7B reaches 82.5 POISE after SFT and 88.1 after PACT, a gain of 5.6 points close to the 5.9 points on Qwen2.5-1.5B-Instruct. Specifically, we address this from two aspects below:

(1) Failure Mode under the Shared Configuration. After SFT under the shared configuration, Qwen3-1.7B intervenes on 79.4% of the situations that warrant silence, consistent with its Acc-P of 56.1 in Tab. 4, against 16.0% for Qwen2.5-1.5B-Instruct (Tab. 22), while both rarely miss a warranted intervention (8.5% vs. 6.7%). PACT still adds 10.3 Pair-C. points on this checkpoint, more than the 8.2 points on Qwen2.5-1.5B-Instruct.

(2) Convergence and Configuration. Under each configuration in Table R4, validation POISE on a held-out 10% of the training split changes by less than one point from the third to the fifth SFT epoch, so the two additional epochs do not close the gap. The shared configuration keeps every SFT setting identical across backbones, including the chat template and the learning rate. Qwen3-1.7B, however, is a hybrid reasoning model with a non-thinking chat template. We have retrained it with this template in both SFT and evaluation and then selected its learning rate on the same held-out split, with all other settings unchanged. All rows use the final SFT checkpoint, and Table R4 below reports the results.

Table R4. SFT and SFT+PACT on POISE under the shared and the backbone-specific SFT configurations.

BackboneSFT configurationValidation POISE, epoch 3 / 5SFT POISESFT FD ↓SFT+PACT POISEGain
Qwen2.5-1.5B-InstructShared82.4 / 83.083.116.089.0+5.9
Qwen3-1.7BShared41.6 / 42.342.179.446.6+4.5
Qwen3-1.7BNon-thinking chat template77.2 / 77.877.924.384.6+6.7
Qwen3-1.7BNon-thinking chat template, learning rate 2×10⁻⁵82.0 / 82.682.517.288.1+5.6

The non-thinking chat template recovers most of the gap, and the selected learning rate closes most of the remainder. Under this configuration, SFT+PACT also exceeds SFT+Outcome (84.4) and SFT+Judge (86.1) on Qwen3-1.7B.

PACT therefore retains its gain on both backbones with the same reward, grouping, and RL hyperparameters. We have added this analysis and Table R4 to App. B.6, and the backbone-specific SFT settings to App. B.3 of the revised manuscript.


We hope this fully addresses your questions and comments, and remain happy to engage in further discussion for the rest of the rebuttal period.

Reviewer_TLrhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

There are two main pain points here: it’s hard to get proper supervision for proactive wearable‑agent systems, and GRPO training suffers from weak advantage signals when all samples in a group share identical rewards. This paper proposes PACT to tackle these issues. The framework uses executable rules to generate supervision labels, construct counterfactual pairs, and compute rewards. It also groups responses from paired inputs when calculating GRPO advantages. The authors introduce POISE to evaluate intervention decisions, tool selection, and assistance delivery, and conduct experiments on POISE and CAB.

Critical strengths

Meaningful problem and substantial effort. The paper considers not only whether assistance should be provided, but also which device to use, how to deliver it, and when to act. The dataset construction, rule checking, and counterfactual pairs reflect substantial work on the overall system.

Clear improvements on POISE. For Qwen2.5-1.5B, the POISE score improves from 83.1 with SFT to 89.0 with PACT, while pair consistency increases from 78.7 to 86.9. The authors also conduct ablation studies by disabling individual reward components and comparing the results, providing additional analysis of the reward design.

Useful dataset. POISE contains 8,559 decisions across 12 scenarios, with intervention and non-intervention examples and counterfactual pairs. These pairs help test whether a model changes its decision when a key condition changes, making the dataset a useful resource for future work on proactive assistance.

Critical weaknesses

The novelty needs further explanation. Counterfactual data augmentation, contrast-set evaluation, and rule-based supervision have been widely studied. "Label derivability" mostly describes how the dataset is built. I think the main contribution seems to be the integration of existing components, and the authors should explain more clearly what is new in the method itself.

The benefit of paired GRPO is not directly tested. The paper compares different rewards and hyperparameters, but does not clearly report a comparison between standard within-input grouping, random cross-input grouping, and counterfactual pairing under the same reward. It is therefore unclear how much of the improvement comes from pairing itself. I suggest prioritizing these grouping comparisons. If possible, please also compare with other reinforcement learning methods, such as DAPO.

The evidence for a broader performance advantage is still limited. The gains on POISE are encouraging, but PACT does not consistently outperform the other RL rewards on CAB. I suggest adding comparisons with recent relevant methods and evaluations on more related benchmarks to verify its advantages for proactive decisions, tool selection and assistance delivery. Besides, I also think the large performance gap between two similarly‑sized models after PACT training deserves further explanation.

The writing could be simpler. The overall logic is understandable, but some descriptions use complex terms and indirect wording for relatively simple operations. I suggest supplementing a concrete example walking through rules, labels, rewards and grouping together to help better understand the work.

Questions for authors
  1. Do Outcome and Judge in Table 4 also use counterfactual pairing? With the reward and sampling budget fixed, how much does pairing improve over standard grouping and random grouping?

  2. The appendix describes how intervention labels and reference plans are generated, but how are the reference proactivity scores from 1 to 5 assigned? Are they computed using explicit rules? Could the authors provide a complete example?

3 comments
AuthorsOpenReview ↗

Authors' Rebuttal (1/2)

Thank you for providing us with your valuable feedback. We have carefully considered your comments and would like to address them as below.


W1 [What is new in the method itself]

PACT uses counterfactual pairs verified by rule execution as the unit of advantage normalization in group-relative RL, unlike the counterfactual and rule-based methods below. Specifically, we address this from three aspects below:

(1) Counterfactual Augmentation and Contrast Sets. Kaushik et al. (2019) train on inputs that annotators edit and relabel, and Gardner et al. (2020) use such edits as contrast sets that evaluate local decision boundaries. PACT verifies the label flip of each counterfactual by re-executing the predicates and pools the rollouts of the pair into one optimization group, so the advantage contrasts situations that differ only in the decisive variable (Sec. 3, Eq. 4).

(2) Rule-Based Supervision and Verifiable Rewards. Rule-based rewards in mathematics and code check a final answer, as in Shao et al. (2024) and Guo et al. (2025). PACT re-executes the specification that produced the label to verify six components of a proactive decision (Eq. 3, App. B.3).

(3) Role of Label Derivability. Beyond data construction, label derivability supports the reward and the grouping at training time. The approaches of (1) rely on annotators to judge each counterfactual, whereas here its label and reward target follow from execution. PACT keeps the GRPO objective, so its grouping can be isolated and combined with other algorithms. The grouping adds 3.4 POISE points on its own and also improves DAPO (Tables R1 and R2 below).


W2 & Q1 [Pairing in Tab. 4, grouping under a fixed reward, and DAPO]

Yes, on POISE the Outcome and Judge rewards in Tab. 4 also use counterfactual pairing, and under the fixed PACT reward and 80 rollouts per step, pairing improves POISE by 3.4 points over within-input grouping and by 3.2 points over random grouping. The three GRPO variants differ only in their reward definitions (Sec. 5), so Tab. 4 isolates the reward, while Table R1 below isolates the grouping. Specifically, we address the remaining questions from two aspects below:

(1) Grouping under a Fixed Reward. We have added a grouping ablation with two training epochs per run. Random cross-input grouping joins two unrelated inputs, and its opposite-label variant joins two inputs of the same scenario with opposite labels. Table R1 below reports the results and the share of zero-variance groups, in which all rollouts receive the same reward.

Table R1. Grouping ablation on Qwen2.5-1.5B-Instruct with the PACT reward, the same SFT initialization and training data, and 80 rollouts per step.

GroupingPOISEAcc-PPair-C.Tool-F1Chan.-F1Zero-variance groups (%)
None (SFT)83.188.778.781.483.7n/a
Within-input (standard GRPO)85.691.082.184.085.441.7
Random cross-input85.891.483.083.685.19.6
Random cross-input, opposite labels86.692.084.184.385.95.8
Counterfactual pairs (PACT)89.093.386.986.889.07.1

Random pairing adds only 0.2 points over standard GRPO, and opposite-label pairs, which leave the fewest zero-variance groups, remain 2.4 points below counterfactual pairs. Reward differences across inputs alone therefore do not account for the gain, which requires pairing situations that differ only in the decisive variable.

(2) Comparison with DAPO. DAPO (Yu et al., 2025) discards zero-variance groups, samples further inputs, and raises the upper clipping bound. We have applied these mechanisms with the PACT reward to both groupings (Table R2 below).

Table R2. GRPO and DAPO under the setting of Table R1.

RL algorithmGroupingPOISEPair-C.Rollouts per step
GRPOWithin-input85.682.180
DAPOWithin-input86.983.9137
GRPO (PACT)Counterfactual pairs89.086.980
DAPOCounterfactual pairs89.387.586

Pairing also improves DAPO, from 86.9 to 89.3 POISE, so discarding zero-variance groups does not replace controlled pairing, and within-input DAPO remains 2.1 points below PACT. We have added Tables R1 and R2 to App. B.6 of the revised manuscript.


Please see the subsequent Authors' Rebuttal (2/2) for remaining responses.

AuthorsOpenReview ↗

Authors' Rebuttal (2/2)

Continued: Authors’ Rebuttal (2/2)


W3 [CAB results, further benchmarks, and the gap between backbones]

Thank you for your comments. The CAB columns of Tab. 4 evaluate a reduced variant of PACT, and once PACT is trained in full on POISE, it transfers to CAB and ProactiveBench better than the Outcome and Judge rewards. Specifically, we address this from three aspects below:

(1) Results on CAB. CAB provides 600 training samples with answer-level labels assigned by an LLM, without executable rules, counterfactual pairs, or delivery targets (Tab. 1, Tab. 8). Trained on CAB, PACT therefore keeps only its decision and tool components with within-input groups, the same signals that the Judge reward uses. The rewards are accordingly close on CAB, where the best method varies with the metric and PACT gives the highest Score on Qwen3-1.7B. On POISE, where the full reward and counterfactual groups apply, PACT is best on both backbones.

(2) Further Benchmarks and Methods. The comparisons include single-context adaptations of Inner Thoughts and Proactive Agent (Tab. 3), the latter also on CAB, and Table R2 above adds DAPO. We have also evaluated the POISE-trained models zero-shot on CAB and ProactiveBench (Lu et al., 2025), reported in Table R3 below.

Table R3. Zero-shot results of Qwen2.5-1.5B-Instruct models trained only on POISE.

ModelCAB IndexCAB ScoreCAB Tool-F1ProactiveBench F1ProactiveBench false-alarm rate ↓
Direct (no training)54.52.345.541.763.5
SFT73.031.858.449.235.1
SFT + GRPO, Outcome74.333.059.150.633.4
SFT + GRPO, Judge73.835.160.350.134.2
SFT + PACT76.538.662.754.426.7

PACT lowers the ProactiveBench false-alarm rate by 8.4 points relative to SFT.

(3) Gap between Backbones. A backbone-specific SFT configuration closes most of the gap. Under the shared configuration, Qwen3-1.7B after SFT intervenes on 79.4% of the situations that warrant silence, against 16.0% for Qwen2.5-1.5B-Instruct (Tab. 22). With the non-thinking chat template of Qwen3-1.7B in SFT and evaluation and a learning rate selected on held-out training data, the model reaches 82.5 POISE after SFT and 88.1 after PACT, a gain close to the 5.9 points on Qwen2.5-1.5B-Instruct.

We have added Table R3 and the backbone analysis to App. B.6 and stated the CAB training protocol in Sec. 5 of the revised manuscript.


W4 & Q2 [A complete example, and assignment of proactivity scores]

Each scenario specification fixes a score for every response and suppression case, and executing it on a state selects the case. Specifically, we address this from two aspects below:

(1) Score Assignment. A response, selected when all triggers hold and no suppression condition applies, carries a score of 3, 4, or 5 by urgency. A suppression case carries a score of 1 when the need is absent, the user is already handling it, or the evidence is unreliable, and 2 when a weaker need remains. A score of 3 or above means intervention (App. A.1).

(2) Complete Example from Airport Flow.

  • Rule. Intervention requires all triggers, including the user's own booking, an electronic pass, boarding within 180 minutes, and an indoor fix within 25 m on the correct floor, and no suppression condition, such as an open airline app.
  • Labels. In s+\mathbf{s}^{+}, the user is in the security queue 157 minutes before boarding. The journey stage selects the response (score 4 at arrival, check-in, and security, 5 during wayfinding and at the gate), so s+\mathbf{s}^{+} receives score 4 with positioning, pass display, and speech on the watch and earbuds. s−\mathbf{s}^{-} changes only whether the booking is in the user's name, as the user is seeing someone off, so a trigger fails, giving score 1 with no action, and the booking owner is the decisive variable V⋆V^{\star}.
  • Reward. The target output of s+\mathbf{s}^{+}, which names this variable as decision evidence, obtains R=1.0R=1.0 on s+\mathbf{s}^{+} and R=0.30R=0.30 on s−\mathbf{s}^{-}. There, only the evidence, safety, and timing components remain, and they check the named variable and the delivery constraints rather than the decision (Eq. 3, App. B.3).
  • Grouping. If all five rollouts on each input produce this output after SFT, within-input groups yield zero advantage. The pair-contrastive group of ten rollouts gives those on s+\mathbf{s}^{+} a positive and those on s−\mathbf{s}^{-} a negative advantage, which discourages acting when the booking belongs to someone else (Eq. 4).

The same rules thus determine the scores, labels, and rewards, and the verified pair defines the group. We have added this example to App. B.3 and the score rules to App. A, and simplified the wording of Sec. 3 in the revised manuscript.


We hope this addresses your concerns and are happy to discuss further.

Thank you for your response and clarifications. After considering the rebuttal, I have decided to retain my original score.

Reviewer_kmo1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents PACT, designed for training proactive wearable language agents on intervention timing, tool selection, and assistance strategy. Supervision is generated via executable scenario rules to calculate rewards in reinforcement learning. Rollouts from counterfactual settings with opposing intervention labels are grouped to preserve reward contrast. A benchmark named POISE is constructed with 8,559 text-rendered decisions across twelve scenarios. The authors perform experiments comparing prompting, fine-tuning, and reinforcement learning, showing progress on POISE and varied improvements on the CAB benchmark.

Critical strengths
  • Designing verifiable wearable proactive agents is useful and interesting.
  • Authors provide new benchmarks and conduct many experiments.
Critical weaknesses
  • After reading the paper, although the paper has an interesting and useful underlying idea, the clear research gap or the practical problem, supervision problem and optimization problem are not clearly separated and clarified. All the problems stated are miss connected.
  • I appreciate the new benchmarking, yet the benchmark now still looks like a structured decision task rather than a real proactive setting. And authors should identify the exact proactive pipeline early in the paper.
  • The evaluation metric selection, averaging various metrics, complicates the setting.
Questions for authors
  • What's the independent benefit of counterfactual grouping? Can the authors provide the ablations of separating different groupings?
  • Could the authors explain the justification behind the intervention rules along with the suppression conditions?
2 comments
AuthorsOpenReview ↗

Authors' Rebuttal (1/2)

Thank you for providing us with your valuable feedback. We have carefully considered your comments and would like to address them as below.

W1 [Separation of the practical, supervision, and optimization problems]

The three problems are distinct and form one chain, in which the executable specification that resolves the supervision problem also supplies the reward contrast that the optimization problem lacks. The practical problem is selective intervention. Many situations warrant no intervention, so a useful policy must act only when the conditions call for assistance. The supervision problem is that a decision label alone does not record these conditions. Imagine a cyclist whose earbuds detect a vehicle behind. The agent should warn when the vehicle closes in fast and stay silent when it follows at a steady distance. The label "warn" records the decision but not the approach rate that produces it, so it can check the final decision of a new output but neither verify its evidence, delivery, and timing nor certify that the steady-following situation changes the label (Sec. 1, Fig. 1a). PACT therefore keeps the trigger and suppression predicates executable, which yields the labels, label-changing counterfactual pairs, and a reward that re-executes the predicates on every rollout (Sec. 3, Eq. 3).

The optimization problem is that after SFT, a policy that warns about every vehicle behind can receive the same reward on all rollouts of an input, which leaves group-relative RL without an advantage. Pairing the fast-closing and the steady-following situations restores the advantage, because the same warning is correct on one half and wrong on the other and therefore receives different decision rewards (App. B.3). The specification thus verifies the pairs, and grouping their rollouts turns them into the optimization signal.


W2 [Realism of POISE and the proactive pipeline]

POISE evaluates the decision step of a wearable agent offline, from a textual description of its sensor context, and derives and verifies the labels from the structured state behind this description. Specifically, we address this from two aspects below:

(1) Proactive Pipeline. A wearable agent continuously senses its user, describes the current context together with the persona and device states, decides through a proactivity score whether to intervene, selects tools, and plans the device, modality, and timing of delivery. For a given context, false and missed interventions are determined at the decision step. Like CAB and ProactiveBench (Tab. 1), POISE evaluates this step offline on the described context, and PACT trains it by re-executing the specification on each output (Fig. 2).

(2) Inputs and Outputs. Each prompt describes about 21 signal variables in prose, never the structured state, together with device availability and, for most samples, a user persona (Sec. 3, Tab. 8), and 3,076 of the 8,559 samples are grounded in five public sensor recordings (Tab. 7). Suppression conditions also depend on the recent history of the user, such as repeated alerts in the past week or a recent dismissal. The output is structured because a wearable agent must issue executable tool calls and a delivery plan. On the 1,284 test samples grounded in recordings, SFT+PACT reaches 88.4 POISE against 82.6 for SFT, a gain consistent with the 89.0 against 83.1 on the full split (Tab. 4).

POISE therefore evaluates this decision step on inputs grounded in recordings, user history, and device constraints. We have described this pipeline at the start of Sec. 1 of the revised manuscript.


W3 [Choice and aggregation of metrics]

The POISE score averages four complementary metrics, all of which are also reported separately. Acc-P measures whether to intervene, Tool-F1 what to provide, Chan.-F1 how to deliver it, and Pair-C. whether the decision follows the decisive condition, all on a 0 to 100 scale (Sec. 5, App. A.1). Acc-P and Pair-C. enter separately because Pair-C. credits a pair only when both of its decisions are correct. Always acting reaches 50.0 Acc-P but 0.0 Pair-C. (Tab. 2), so Pair-C. exposes a policy that ignores the decisive condition. Safety and timing are separate requirements on the delivery plan, so MSVR and actionability are reported next to the score rather than averaged into it, together with DVI and RMSE (Tab. 4, Tab. 22). On Qwen2.5-1.5B-Instruct, the four SFT-initialized variants follow the same order on each of the four metrics.

Any positively weighted average of the four metrics therefore ranks these variants identically, with SFT+PACT first.


Please see the subsequent Authors' Rebuttal (2/2) for remaining responses.

AuthorsOpenReview ↗

Authors' Rebuttal (2/2)

Continued: Authors’ Rebuttal (2/2)


Q1 [Independent benefit of counterfactual grouping]

Thank you for this question. We have added a grouping ablation with the same reward and training data and 80 rollouts per step, in which counterfactual grouping improves POISE by 3.4 points and Pair-C. by 4.8 points over within-input grouping.

All runs use 16 inputs with five rollouts each per step and train for two epochs. Random cross-input grouping joins two unrelated inputs, and its opposite-label variant joins two inputs of the same scenario with opposite labels. The row with 10 rollouts per input uses 8 inputs per step and thus matches the group size of pairing. Table R1 below reports the results, including the share of zero-variance groups, in which all rollouts receive the same reward.

Table R1. Grouping ablation on Qwen2.5-1.5B-Instruct with the PACT reward, the same SFT initialization and training data, and 80 rollouts per step.

GroupingPOISEAcc-PFD ↓Pair-C.Tool-F1Chan.-F1Zero-variance groups (%)
None (SFT)83.188.716.078.781.483.7n/a
Within-input (standard GRPO)85.691.013.282.184.085.441.7
Within-input, 10 rollouts per input85.991.212.982.684.285.634.9
Random cross-input85.891.412.483.083.685.19.6
Random cross-input, opposite labels86.692.011.384.184.385.95.8
Counterfactual pairs (PACT)89.093.39.986.986.889.07.1

Larger within-input groups and random pairing add only 0.3 and 0.2 points over standard GRPO. Opposite-label pairs leave the fewest zero-variance groups yet remain 2.4 POISE and 2.8 Pair-C. points below counterfactual pairs. Reward differences across inputs alone therefore do not account for the gain, which requires pairing situations that differ only in the decisive variable. We have added Table R1 to App. B.6 of the revised manuscript.


Q2 [Justification of intervention rules and suppression conditions]

Triggers state when assistance becomes actionable and suppression conditions state reasons to stay silent, with sensor thresholds calibrated against public recordings, contextual thresholds set by the requirements of each scenario, and decisions supported by annotators who did not see the labels. Specifically, we address this from three aspects below:

(1) Triggers and Suppression Conditions. In Airport flow, for example, the triggers require the user's own booking, an electronic pass, boarding within 180 minutes, and an indoor fix within 25 m on the correct floor. Consistent with the mixed-initiative principles of Horvitz (1999) and the receptivity study of Mehrotra et al. (2016), suppression conditions cover four reasons for silence. The need may be absent, as when boarding is still hours away, or the user may already be handling it, as when the airline app is open. The evidence may be unreliable, as when the fix lies on the wrong floor, or the interruption cost may be too high, as after a recent alert on the same condition in Road crossing.

(2) Calibration of Thresholds. The talk-pacing suppression threshold of 20% sits at the 90th percentile of within-speaker rate variation in LibriSpeech (19.8%), and the trigger of 25% lies above it (Tab. 9). The 25 m usability gate on indoor positioning lies above the 90th percentile of the UJIIndoorLoc positioning error (19 m, median 5.5 m), and the child-route threshold of three standard deviations sits at the 90th percentile of the distance from frequented locations in GeoLife. Heading enters a trigger only when its jitter is below 35°, because GeoLife bearing jitter reaches 68° at the 90th percentile. We have listed all 63 numerical conditions of the twelve scenarios with their rationale in App. D of the revised manuscript.

(3) Validation by Annotators. Without seeing the derived label or the decisive variable, annotators agreed with the derived decision, with a score within one level, in 88.6% of their ratings (α = 0.73) (Tab. 11). After seeing both halves, 87.8% of their ratings judged the pair to differ only in a variable sufficient to reverse the decision.

The rules therefore encode documented reasons for acting and for staying silent.


We hope this fully addresses your questions and comments, and remain happy to engage in further discussion for the rest of the rebuttal period.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Reviewers, AC, SAC, and PC,

Thank you for your time and effort in reviewing and handling our paper. We have posted point-by-point responses to each reviewer and summarize the main points below.


Paper Overview and Strengths

[Paper Overview]: This paper proposes PACT, a training framework for wearable language agents that must decide unprompted whether to intervene, what assistance to provide, and how to deliver it. PACT keeps the decision rules executable after data construction, so that they derive supervision and label-changing counterfactual pairs, verify each policy output as a reward, and group rollouts across each pair to preserve reward contrast when within-input rollouts agree. We further introduce POISE, a benchmark of 8,559 wearable decisions across twelve scenarios.

[Strengths]: We are encouraged that the reviewers find proactive assistance for wearable agents a useful and important problem (kmo1, TLrh, M8mD), consider executable rules a coherent and inspectable basis for supervision, counterfactuals, and rewards (M8mD), regard POISE as a new (kmo1) and useful (TLrh) benchmark that tests decisions on counterfactual pairs (TLrh, M8mD) and evaluates tools, delivery, safety, and timing (M8mD), and recognize the substantial work on the dataset and rules (TLrh), the clear gains on POISE (TLrh), the reward-component ablations (TLrh, M8mD), and the extensive experiments (kmo1).


Additional Experiments for Concerns

  • [Counterfactual Grouping] (kmo1, TLrh, M8mD): Under the same reward, data, and 80 rollouts per step (Table R1 for kmo1 and TLrh, R3 for M8mD), counterfactual grouping exceeds within-input grouping by 3.4 POISE and 4.8 Pair-C. points, whereas random pairs with opposite labels, which leave the fewest zero-variance groups, remain 2.4 points lower.
  • [Comparison with DAPO] (TLrh): Pairing also improves DAPO from 86.9 to 89.3 POISE with 86 rather than 137 rollouts per step (Table R2 for TLrh).
  • [Rule Execution Baselines] (M8mD): Rule execution on states extracted by a model of equal size (Table R1 for M8mD) trails PACT by 7.4 POISE points, with a misread decisive variable in 71% of its failed pairs, and a 671B extractor stays below PACT on POISE at about 20 times the latency through its API.
  • [Zero-Shot Transfer] (TLrh, M8mD): Trained only on POISE (Table R3 for TLrh, R2 for M8mD), Qwen2.5-1.5B-Instruct with PACT leads all evaluated models on every CAB and ProactiveBench metric, cutting the ProactiveBench false-alarm rate of SFT by 8.4 points.
  • [Gap between Backbones] (TLrh, M8mD): The gap arises in SFT, where Qwen3-1.7B intervenes in 79.4% of situations warranting silence. With its non-thinking template and a tuned learning rate (Table R4 for M8mD), it reaches 82.5 POISE after SFT and 88.1 after PACT.

Summary of Responses to Reviewers' Questions

  • [Problem Formulation and Pipeline] (kmo1): The three problems are distinct but linked, as the executable specification that resolves supervision also supplies the reward contrast that optimization lacks. POISE evaluates the decision stage of the wearable pipeline now described in Sec. 1, with 3,076 samples grounded in five public sensor recordings, and PACT retains its gain over SFT on the 1,284 such test samples.
  • [Metric Aggregation] (kmo1): All four averaged metrics are reported separately, and the SFT-initialized variants of Qwen2.5-1.5B-Instruct rank identically on each, so any positive weighting ranks SFT+PACT first.
  • [Rules and Proactivity Scores] (kmo1, TLrh): Suppression conditions encode four documented reasons for silence, sensor thresholds are calibrated against public recordings, and blind annotators agree with the derived decision in 88.6% of ratings (kmo1). Each specification fixes the score of every case, and a worked Airport flow example connects rules, labels, rewards, and grouping (TLrh).
  • [Novelty] (TLrh): Unlike annotator-edited counterfactuals and final-answer rewards, PACT uses rule-verified counterfactual pairs as the unit of advantage normalization, so one specification supplies the labels, reward, and groups.
  • [Reward and Grouping in Tab. 4] (TLrh, M8mD): On POISE, the Outcome and Judge rewards also use counterfactual pairing, so Tab. 4 isolates the reward and the new ablation the grouping. On CAB, which lacks executable rules and pairs, PACT keeps only decision and tool rewards within-input groups, hence the close results (TLrh).

Revisions During Rebuttal

Revision are highlighted in blue in the revised PDF: the proactive pipeline (Sec. 1), the rule predicates, the role of pairing, and simplified wording (Sec. 3), the CAB protocol and a summary of new analyses (Sec. 5), the score assignment (App. A) and calibrated thresholds (Tab. 9), the worked example and backbone-specific SFT settings (App. B.3), all new experiments (Tabs. 26 to 30, App. B.6), and the rationale of all 63 rule thresholds (App. D).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.