- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies a task-identifying problem in meta-reinforcement learning. The authors point out that the task-identifying accuracy of encoder does not only reflect the information in the data, but also could come from leakage in data collection strategy. Therefore, the authors propose FLAT, a framework aiming to build an upper bound for task-identifying accuracy with statistical guarantee. Specifically, it divide a collected log into three parts: common, calibrated and unknown. The key idea in FLAT is to stop auditing before the first unknown factor and assume the subsequent factors can reveal the real task. The authors then provide three methods to calculate the accuracy upper bound based on different information we have when auditing. The authors further validate their claim in experiments.
Critical strengths
The authors identify the problem that task-identifying accuracy of encoder does not only reflect the information in the data, but also could come from leakage in data collection strategy. And propose a framework for auditing it.
Critical weaknesses
Since the theory part is relatively standard, I am a little bit worried about the significance of this paper. It may be hard for real data to satisfy the assumption in the paper. We may not know the mechanism generating the data.
Questions for authors
- Could the authors further clarify the practical significance of the proposed method? Specifically, it would be good if the authors could provide a concrete example.
3 comments
Thank you for the clarification and examples. The explanation and experiments results are helpful. But I think the world model experiment is not enough to support the framework usefullness under unknown mechanism. The strong result depends on the world model knowing the simulation structure. Overall, I decide to keep my score.
Thank you for the follow-up. We agree. The simulator-structured result assumes a known simulator family, as stated in Appendix G.3, and our experiments do not show that learned likelihoods are reliable for unknown task-dependent dynamics.
We would like to clarify one point of scope. FLAT needs likelihoods only for factors that depend on the task. A factor whose law is the same under every candidate is declared COMMON and cancels from every comparison, so its numerical form is never needed. What is required is the qualitative fact that the factor does not depend on the task. This is usually fixed by the benchmark definition, and a wrong declaration of this kind can be detected by the diagnostic probes of Appendix C.4.
In reward-varying benchmarks such as PointRobot and HalfCheetah-Vel, the physical transition is COMMON. All certified collection-protocol results in Section 4.2 therefore use no transition model. They still require the collection-policy likelihoods, which the dataset creator holds (Table 1). The shared-policy logs reach the exact prior-level limit with no likelihood at all. Unknown task-dependent dynamics remain a limitation; dynamics that are unknown but task-invariant are not.
We also note that requiring only the simulator family, rather than the exact dynamics, already broadens where FLAT applies. Dynamics-varying OMRL benchmarks are generated from known simulators with randomized parameters, so this setting covers a large class of existing benchmarks, even though the results there are estimates rather than certificates.
Thank you for asking for a concrete use case and for raising concerns about the significance and practical assumptions of an audit built from established information-theoretic ingredients.
Direct question
Q1. Could the authors clarify the practical significance of FLAT with a concrete example?
To illustrate the practical value of FLAT, consider a researcher evaluating a task encoder on an offline meta-RL dataset. Suppose a FOCAL-style identifier achieves an accuracy of 0.581 on reward-free PointRobot contexts. From this number alone, it is difficult to tell whether the encoder has failed to recover available task information or whether the logged data simply do not contain enough information for better identification.
FLAT helps answer this question by providing a certified reference for the best achievable identification accuracy on the same fixed contexts. In our experiments, FLAT places this limit within [0.790, 0.800], revealing at least a 20.9-percentage-point gap between the learned identifier and the information available in the log. This tells the researcher that there is substantial room to improve task identification without necessarily changing the dataset.
FLAT can also help researchers understand how data collection affects this limit. To investigate this, we added experiments comparing different checkpoint-based collection protocols on the same 100 PointRobot tasks. At H=20, the certified limit changes from [0.993, 1.000] for final actors to [0.544, 0.555] for early snapshots. Moreover, even with the same checkpoint pool and number of episodes, independently sampling checkpoints for each episode gives [0.991, 1.000], compared with [0.893, 0.911] when one checkpoint is shared across the context.
Together, these examples illustrate how FLAT can help researchers evaluate encoder limitations, quantify the information available in offline contexts, and understand the effects of collection and context-construction protocols (Section 4.2; Appendix D).
Additional weaknesses
Standard theory. We agree the Blackwell and KL/TV ingredients are established. Our contribution is the complete factor declaration, conservative stop-and-reveal audit, calibrated certificate interface, and certified comparison between a learned encoder and the best achievable identifier on a fixed log—not a new mathematical inequality about OMRL, and we did not claim any theory either.
Applicability to real-world datasets. We acknowledge that real-world offline datasets may lack the collection provenance or calibrated likelihood information needed for an informative certified upper bound. However, this does not necessarily prevent FLAT from being used as an approximate diagnostic. When exact likelihoods are unavailable, behavior cloning (BC) and world models (WM) can be fitted to task-labeled offline trajectories to estimate candidate action or transition likelihoods. These estimates can then provide potentially useful reference values for task-identification evaluation, although they are not certified upper bounds without additional model-error guarantees.
To investigate this possibility, we added experiments comparing FLAT values computed from fitted BC/WM models against exact-likelihood references (Section 4.4; Appendix G). In our controlled experiments, some fitted models produce estimates close to the exact-likelihood references, although their accuracy depends on the model and available data. For example, goal-conditioned BC trained with 1,000 episodes per task gives Real/Model values of 0.514/0.515, compared with the certified PointRobot bracket [0.509, 0.519]. On stochastic Walker, a simulator-structured WM gives 0.804/0.833 against an exact-likelihood reference near 0.810.
We also report cases where fitted models are inaccurate, particularly with limited data. Thus, the additional experiments demonstrate the potential usefulness and limitations of approximate FLAT diagnostics. They do not establish reliable certification or estimation for arbitrary real-world datasets, including human demonstrations, which remain untested.