- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Tiara, an offline in-context reinforcement learning framework that addresses the mismatch between varying task identifiability across context prefixes and uniform action supervision in DPT-style methods. Tiara introduces a causal context encoder with auxiliary task-ID supervision and uses task-classification correctness as a prefix-level reliability signal to reweight action prediction losses. The task classifier and reweighting are used only during training, while deployment adapts to unseen tasks solely through interaction context. Experiments across navigation, locomotion, and manipulation benchmarks show improvements over several offline ICRL baselines in most settings.
Critical strengths
- Motivation is intuitive and targets a concrete mismatch in DPT-style offline ICRL: different prefixes contain different amounts of task information but receive uniform action supervision.
- The method is simple and cheap. The per-sample normalization preserves the total action-loss scale, and the weighting scheme reduces to uniform supervision when , making it a clean drop-in modification. The task classifier and reweighting are train-only, so deployment requires no task IDs and adds no inference cost beyond the encoder–decoder split.
- The evaluation covers discrete navigation, locomotion, and manipulation, plus a higher-dimensional Humanoid-Dir task, against a reasonably broad set of baselines.
Critical weaknesses
- The central reliability signal in Eq. (9) is based on binary task-classification correctness. However, correctness is not equivalent to identifiability: an intrinsically ambiguous prefix may still yield a coincidentally correct argmax, while a nearly correct but slightly misclassified prediction is treated as fully unreliable. This conflates the information content of the prefix with the current state of the learned classifier. The paper would benefit from comparing binary correctness against confidence-, margin-, or entropy-based alternatives and from directly analyzing how these signals evolve with prefix length.
- Task identifiability does not necessarily imply action ambiguity. Multiple tasks that remain indistinguishable under the same prefix may nevertheless prescribe the same optimal action at the query state. In such cases, the task is ambiguous but the action supervision is still fully valid. Since the method reweights an action-prediction loss, the paper should justify why task-ID uncertainty is the appropriate surrogate for action ambiguity and ideally demonstrate a strong empirical relationship between the two.
- The paper motivates Tiara by arguing that ambiguous prefixes should not be strongly supervised toward a unique task-dependent action. However, Eq. (5) applies exact task-ID cross-entropy supervision uniformly to every prefix, including those that may be intrinsically non-identifiable. This appears partially at odds with the motivation and may encourage artificial task separation from insufficient evidence.
- The current ablations do not fully isolate the contributions of the decoupled architecture, auxiliary task supervision, and reliability-guided weighting. In particular, the “w/o Encoder” variant changes both the model architecture and the weighting mechanism, making it difficult to attribute the observed gains. Architecture-matched controls with uniform, position-based, and reliability-based weighting would provide a cleaner evaluation of the proposed mechanism.
Questions for authors
- What fraction of prefixes have over training, and what fraction of samples contain both correct and incorrect prefixes such that Eq. (11) actually produces non-uniform weights? Reporting these statistics by training step and prefix length would clarify how active the proposed mechanism is in practice.
- Tiara assumes that a prefix that cannot reliably identify the task should receive weaker action supervision. However, task ambiguity does not necessarily imply action ambiguity, since multiple indistinguishable tasks may still require the same action at the query state. Can the authors empirically verify that prefixes marked as unreliable indeed exhibit higher target-action disagreement?
- How does binary correctness compare with continuous reliability signals such as the probability assigned to the true task, classification margin, or predictive entropy under the same architecture?
- Is the task classifier relying on behavior-policy artifacts rather than task evidence? Could the authors evaluate task identification on contexts collected by a shared or different behavior policy, or report task-ID accuracy using only versus the full ?
- Could the authors provide architecture-matched ablations comparing uniform, position-based, confidence/entropy-based, and binary reliability weighting, together with a capacity-matched DPT? Sensitivity analyses for and would also be helpful.
4 comments
Reviewer jZq8: are you convinced by the rebuttal, particularly how classifier correctness is related to identifiability?
The rebuttal addresses most of my main concerns, particularly through the new analyses on action disagreement, alternative reliability signals, architecture-matched ablations, and sensitivity studies. These additions substantially strengthen the empirical support for the proposed weighting mechanism. I still have some reservations about the limited novelty and the possibility of behavior-policy confounding, which is not fully ruled out by the current experiments. Overall, however, I am more convinced than before and would be inclined to raise my score from 2 to 3.
We sincerely thank you for the careful reconsideration and the willingness to raise the score. We are glad that the new analyses addressed the main concerns, and we greatly appreciate the constructive suggestions, which have substantially strengthened the paper.
On behavior-policy confounding, we followed your original suggestion and evaluated the classifier on contexts collected by another task’s behavior policy (App. G). The effect depends on the environment: on Reach, the classifier rarely follows the behavior policy, whereas on Walker-Param it often does, and there a different policy also changes the dynamics observed in the context. We agree that TIARA does not remove this confounding, as it is not designed to; since each training task’s data come from its own behavior policy, this does not affect how guides action supervision during training.
Thank you again for your time and the constructive discussion.
We thank the reviewer for the detailed review. Changes in the revised PDF are marked in blue. We number the Critical Weaknesses W1–W4 and the Questions Q1–Q5 in review order.
W1, Q3: Binary correctness conflates prefix information with classifier state; how does it compare with continuous signals?
A: Thank you for this insightful observation. This is intended: the decoder can only use task information already extracted by the encoder, so what matters is whether the current representation identifies the task. In Sec. 5.3 (Table 2), a frozen classifier (same initialization and budget, but not following the encoder) loses about 40% of the gain on Reach and all of it on Walker-Param, and binary correctness outperforms margin, entropy and even the true-task probability, which is gentler on near-misses, in both environments. For a coincidentally correct argmax on an ambiguous prefix, Proposition 1 (Sec. 3.3) shows that the learned target remains a posterior mixture.
W2, Q2: Do prefixes marked as unreliable show higher target-action disagreement?
A: Thank you for this suggestion. Yes (App. G). On Walker-Param, we approximate each task's target action at the query state with a task-conditioned behavior-cloning network and compare misidentified and correct prefixes within the same task and position (three seeds). Misidentified prefixes show 3–8× higher disagreement among the candidate tasks' target actions at prefix lengths 1–5 (AUROC 0.76–0.85), and higher action error. Fewer than 6% of them are assigned a task with a close target action; by Proposition 1, such prefixes only lose gradient share, without biasing the target.
W3: Is applying task-ID cross-entropy to every prefix (Eq. 5), including non-identifiable ones, at odds with the motivation?
A: Thank you for raising this point. We respectfully disagree. Cross-entropy is a proper scoring rule minimized by the task posterior, which is not one-hot on ambiguous prefixes, so it does not force commitment. The action head instead makes a point prediction, which is where uniform supervision is problematic. Empirically, the classifier is calibrated on short prefixes (App. G).
W4, Q5: Architecture-matched ablations, capacity-matched DPT, and sensitivity to α, λ.
A: Thank you for these suggestions. "w/o Encoder" changes both architecture and weighting by necessity, not from incomplete decoupling: ρ comes from the encoder's classifier, so without the encoder there is no identification outcome to weight by, and a position schedule is the closest substitute. To separate the two factors:
Q1: What fraction of prefixes have ρ=1 over training, and how often does Eq. (11) actually produce non-uniform weights?
A: Thank you for this question. App. G (Fig. 9, Table 14) reports both by training step and prefix length. The fraction of prefixes with ρ=1 grows with prefix length and over training, and samples containing both correct and incorrect prefixes remain common throughout training, so Eq. (11) is active in practice. As identification improves, the misidentified prefixes concentrate at the start of the context: the weighting acts most strongly early in training, on the short prefixes that the representation cannot yet identify. Removing it (α=1) lowers the return substantially (App. F).
Q4: Does the classifier rely on behavior-policy artifacts rather than task evidence?
A: Thanks for this important question. Behavior-policy confounding of task inference is a known issue in context-based RL, since the (s,a,s′,r) pairs in a context reflect both the task and the policy that collected them, and dedicated methods have been developed specifically to address it. This issue is orthogonal to our focus: TIARA does not aim to learn a policy-invariant task representation, but uses the classifier only during training to decide how strongly each prefix supervises the action. Our (s,a) versus full-transition comparison (App. G) shows that (s,a) alone carries some task information, but the environment’s feedback contributes clearly more, so the signal mainly reflects task evidence. The classifier is discarded at test time, where TIARA acts on its own contexts and achieves the gains in Table 1.
Thank you again for the helpful comments.