acceptodds
Under review as a conference paper at ICLR 2027

MIRAGE: Benchmarking Intent Identifiability under Partial Observation through Counterfactual Multi-Agent Reasoning

Abstract

Intent inference from partial observations underpins autonomous systems, multi-agent interaction, and situation awareness, yet existing formulations assume that an observed trajectory determines a unique latent intent. This assumption fails in settings where different intentions can generate similar behaviors: a movement toward a region may correspond to an attack, a reconnaissance pass, or a deliberate feint, and single-label evaluation cannot distinguish poor reasoning from intrinsically ambiguous evidence. We introduce MIRAGE, a counterfactual benchmark for intent identifiability. MIRAGE generates matched counterfactual trajectories by intervening on latent intent while keeping scenario conditions fixed. We formalise the counterfactual equivalence set (CES), the set of intents compatible with an observation prefix at each step, using an explicit observation distance and a calibrated indistinguishability threshold , released as a criterion-indexed family and reported under its strictest member. We further introduce ShenJi, an evidence-aware set-valued reasoner that maintains multiple hypotheses and contracts them as discriminative evidence emerges, with contraction enforced by a cardinality regulariser rather than observed post hoc. Across ten runs, 38.7% of trajectory steps remain ambiguous under CES (88.5% within the evaluation window, a lower bound). ShenJi preserves the commanded intent in 94.5% of ambiguous windows while reducing premature exclusion to 0.006, compared with 0.090 and 0.157 for deep-ensemble and conformal baselines. However, discrimination on resolved windows remains below chance level. Selective abstention and adaptive conformal prediction cannot recover the missing evidence, indicating that the observation contract, rather than estimator capacity alone, limits identifiability.

Primary area:
datasets and benchmarks
Keywords:
ntent identifiability, partial observation, counterfactual reasoning, set-valued prediction, multi-agent behavior understanding

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_k8fXOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

When we observe only a part of an agent's thought process, the final intent is not clear yet. Existing methods may force the model to commit to a single intention, while the trajectory so far might be consistent with more than one. Given this motivation, this paper provides a benchmark for identifying a set of possible intentions rather than a single one after observing a chain of thought.

Based on this motivation, the authors introduce MIRAGE, a benchmark that contains trajectories matched under running them with different intentions, and ShenJi, which uses MIRAGE to identify a set of intentions consistent with the current behaviour pattern rather than a single one.

However, I have some concerns about this paper, which I will describe in more detail below, which makes me believe that it is not ready for acceptance in its current form.

Critical strengths

1- I think that the paper genuinely identifies a weakness in how current intent-recognition systems work. Whenever we have partial observations, they can be consistent with more than one intention so the set-valued output of ShenJi is an interesting extension.

Critical weaknesses

1- The proposed methodology seems heavily dependent on the chosen threshold for distinguishability. The results shown in the paper range from very distinguishable to not distinguishable at all, depending on how the threshold is chosen (strict threshold vs. from held-out data vs. from benchmark data)

2- I think that the formula in Proposition 1 might be wrong. If intentions only begin to differ from a step t_0, my calculations show that the distance exceeds the threshold when t>δ(t0−1)δ−ϵot>\frac{\delta(t_0-1)}{\delta-\epsilon_o}, not when t>δ(t0−1)δ−ϵot>\frac{\delta(t_0-1)}{\delta-\epsilon_o} as reported in Proposition 1. The explanation for this is that when the previous time points in which there is no distinction between the intentions are present in the average, then they are going to dilute the signal, and there are many easily seen counterexamples for the currently reported formula.

3- Proposition 1 does not necessarily show that the candidate set always becomes smaller. In fact, a new intention can enter the set at a later time and this is not covered by the current formulation.

4- The human annoted results might show that some of the claims are overstated, as it is mentioned that some evaluated examples have only one possible intention, whereas none of the human annotators chose a single intention for any of the examples. This is not necessarily right or wrong, it is just the claiming needs to be adjusted and this mismatch needs to be explained.

5- It seems like ShenJi is behaving poorly when there is one possible intention. This can be seen in Fig. 1 for example, with an accuracy of 0.187.

6- Overall, I cannot see a clear advantage for ShenJi in the reported experiments. For instance, as mentioned by the paper itself, predicting every possible intention in a set will receive a higher Set-F1 score than ShenJi, and at matched coverage, the conformal prediction method produces smaller sets than ShenJi.

Questions for authors

Please refer to all the weaknesses pointed out above and explain/justify/correct them. Thanks a lot!

8 comments
AuthorsOpenReview ↗

Response to W1: Sensitivity of CES to the Indistinguishability Criterion

We sincerely thank the reviewer for this insightful and important observation. We fully agree that intent identifiability is highly sensitive to the indistinguishability criterion. Your comment has helped us recognize the importance of making this dependence explicit, rather than presenting identifiability as an intrinsic property independent of the observation criterion.

We have clarified this issue in the revised manuscript.

1. Explicitly defining criterion-dependent identifiability (Page 3, Lines 139–150).

We now emphasize that CES is a family indexed by the observation distance and indistinguishability threshold, rather than a unique, criterion-independent ground truth. Different criteria can produce substantially different identifiability boundaries, and their results must therefore be interpreted separately.

2. Quantifying criterion sensitivity (Page 8, Lines 402–416; Appendix C.3, Pages 19–20).

Our analysis shows substantial differences across three settings:

  • Strict exact-signature criterion: 236/288 pairs become distinguishable, with median onset at 0.30 of the trajectory.
  • Held-out canonical calibration: ϵo=0.0125\epsilon_o=0.0125, with 76/288 distinguishable pairs.
  • In-corpus calibration:ϵo=0.0185\epsilon_o=0.0185, with only 14/288 distinguishable pairs and median onset at 0.93.

The two calibrated thresholds differ by approximately 1.48-fold. Their respective calibration collections contain 213/72 and 392/144 same-intent/different-intent pairs, with false-merge rates of 45.8% and 95%. These results confirm the reviewer's concern that identifiability conclusions depend strongly on the chosen criterion.

3. Clarifying the role of the released ground truth.

We use the strict exact-signature criterion to define the earliest observable distinguishability boundary. It provides a reproducible reference and a lower bound on ambiguity within the considered criterion family, but should not be interpreted as a universally optimal or practically calibrated threshold. For applications concerned with observational indistinguishability under execution variability, a calibrated criterion is more appropriate.

We fully recognize that selecting and transferring suitable thresholds remains an important research challenge. We intend to investigate more robust calibration procedures across different observation settings and execution conditions.

We are genuinely grateful for this suggestion. It has helped us substantially improve the interpretation of MIRAGE and clarify that the framework's contribution lies in making criterion-dependent identifiability measurable and reproducible, rather than claiming a universally fixed identifiability boundary.

AuthorsOpenReview ↗

Response to W2: Correction of the Time Bound in Proposition 1

We sincerely thank the reviewer for identifying this important mathematical error. We fully agree with your analysis, particularly your observation that the earlier time steps without distinguishable evidence dilute the accumulated discrepancy. After carefully re-examining the original proof, we confirmed that the reported time bound was incorrect. Your rigorous observation has been invaluable in correcting our theoretical formulation.

We have corrected Proposition 1 and its proof (Page 4, Lines 205–215; Page 20, Lines 1062–1074).

Under the assumption that d(s)≥δ>ϵod(s)\geq\delta>\epsilon_o for all s≥t0s\geq t_0, the prefix-mean discrepancy satisfies

=\frac{1}{t}\sum_{s=1}^{t}d(s) \geq\frac{\delta(t-t_0+1)}{t}.$$ Therefore, the observation distance is guaranteed to exceed the indistinguishability threshold whenever $$t>\frac{\delta(t_0-1)}{\delta-\epsilon_o}.$$ This corrected bound explicitly accounts for the preceding observations that dilute the accumulated evidence, addressing the issue you identified. **We have also revised Proposition 1 from "Monotone contraction" to "Eventual exclusion"**, reflecting the precise guarantee established by the corrected proof. The result provides a sufficient time bound for eventual exclusion under sustained divergence, rather than a guarantee of monotonic contraction. Importantly, this correction does not change the definition of MIRAGE's released exact-signature CES, whose nestedness follows from its construction rather than Proposition 1. Instead, it corrects the theoretical justification and establishes a more rigorous boundary for our claims. We are genuinely grateful for your careful mathematical scrutiny. Your correction has substantially improved the theoretical accuracy and scientific rigor of our work.
AuthorsOpenReview ↗

Response to W3: Correction of the Monotonic Contraction Claim

We sincerely thank the reviewer for identifying this important theoretical flaw. We fully agree that Proposition 1, as originally stated, does not guarantee monotonic contraction of the CES. Your observation that an excluded intent may re-enter the candidate set prompted us to carefully re-examine the underlying assumptions and revise our theoretical claims.

1. Acknowledging the failure of monotonic contraction.

The original argument incorrectly assumed that the prefix mean of nonnegative discrepancies must increase monotonically. This is not generally true.

For example, let the indistinguishability threshold be ϵo=0.4\epsilon_o=0.4, with per-step discrepancies d(1)=1d(1)=1, d(2)=0d(2)=0, and d(3)=0d(3)=0. The corresponding prefix means are 11, 0.50.5, and 1/31/3. An alternative intent excluded at earlier steps therefore becomes compatible again at t=3t=3. This possibility remains even if sustained divergence begins at a later time.

We acknowledge that the original monotonic-contraction conclusion was incorrect.

2. Correcting Proposition 1 (Page 4, Lines 205–215; Page 20, Lines 1062–1069).

We have replaced the original Monotone contraction proposition with Proposition 1 (Eventual exclusion). The revised result establishes that sustained divergence eventually excludes an alternative intent, but does not guarantee that the CES decreases monotonically at every preceding step.

We have explicitly withdrawn the unsupported monotonicity claim and corrected the corresponding proof.

3. Distinguishing theoretical guarantees from construction-specific properties.

Importantly, the nestedness of the released MIRAGE ground-truth CES follows from its exact-signature construction. Once a counterfactual observation prefix diverges from the reference, that difference remains in subsequent prefixes, preventing the excluded intent from re-entering.

This construction-specific property should not be confused with a general monotonicity guarantee for threshold-based CES.

We have also clarified that ShenJi's reported rank-margin decoder does not enforce strict nestedness. The optional no-reentry projection is not applied in our experiments (Page 7, Lines 368–377; Page 10, Lines 511–516).

We are genuinely grateful for the reviewer's rigorous observation. It has helped us correct an important theoretical overstatement and establish a clearer distinction between eventual exclusion, construction-specific nestedness, and empirical prediction-set behavior. These corrections substantially improve the theoretical accuracy and scientific rigor of MIRAGE.

Reviewer_i6dgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors realize that in some partial observation scenarios, multiple intentions may result in similar behaviors, while the existing intent inference assumes that a trajectory corresponds to only one intention. To address this issue, the paper proposes a counterfactual benchmark MIGARE for the identifiability of intentions under partial observation. The author also proposed ShenJi, an evidence-aware set-valued reasoning model with explicit contraction regularization. The experiments evaluated MIGARE in five aspects, including ambiguity validity, behavioral reasoning, single-label failure modes, set-valued uncertainty resolution, and temporal contraction.

Critical strengths
  1. The research topic of this paper is very interesting and it makes a meaningful distinction from the traditional single-label intent recognition.

  2. The matching counterfactual design of MIGARE is quite innovative. In particular, the paper distinguishes between latent intent, realized behavior, and partial observation, thereby reducing the risk of directly exposing the true generated labels to the prediction model. Figure 1 is easy to understand.

Critical weaknesses
  1. The monotonic contraction conclusion of Proposition 1 does not hold under the current assumptions. Specifically, there is a problem with the time bound given in Proposition 1 (Monotone contraction). Under the current assumptions, CES is not necessarily monotonic contraction.

  2. The performance of ShenJi proposed in the paper on identifiable samples is too low, which weakens its persuasiveness as the main method. The core components of ShenJi lack sufficient independent ablation experiments.

Questions for authors
  1. Can the authors clarify and rigorously justify the monotone-contraction guarantees in Propositions 1?
  2. According to the author's own definition of CES, observational evidence has already been able to distinguish intentions. However, Shen Ji still cannot effectively utilize this information for prediction. Does this indicate that the current model cannot prove that it possesses sufficient evidence-aware discrimination capability?
3 comments
AuthorsOpenReview ↗

Response to W1: Correction of Proposition 1 and Its Theoretical Scope

We sincerely thank the reviewer for identifying this important theoretical error. We fully agree with your criticism. After carefully re-examining Proposition 1 and its proof, we confirmed that the original time bound was incorrect and that the monotonic contraction claim did not follow from the stated assumptions. Your observation has been particularly valuable in helping us correct the theoretical analysis and clarify the precise guarantees of MIRAGE.

We have substantially revised Proposition 1 and its proof as follows.

1. Correcting the time bound (Page 4, Lines 205–208; Page 20, Lines 1062–1074).

Under the assumption that d(s)≥δ>ϵod(s)\geq\delta>\epsilon_o for all s≥t0s\geq t_0, the cumulative observation distance satisfies

=\frac{1}{t}\sum_{s=1}^{t}d(s) \geq\frac{\delta(t-t_0+1)}{t}.$$ Consequently, an alternative intent is excluded whenever $$t>\frac{\delta(t_0-1)}{\delta-\epsilon_o}.$$ This corrects the erroneous bound in the original Proposition 1. The revised proposition is now titled **"Eventual exclusion"**, reflecting the property actually established by the proof. **2. Withdrawing the unsupported monotonic contraction claim (Page 4, Lines 209–215).** We acknowledge that a prefix-mean distance need not increase monotonically. An excluded intent may therefore become compatible again before its eventual exclusion. The original assumptions guarantee eventual exclusion of an alternative intent under sustained divergence, but do not establish that the entire CES contracts monotonically. We have explicitly withdrawn this stronger claim. **3. Clarifying the implications for MIRAGE.** Importantly, the nestedness of the released MIRAGE ground-truth CES arises from its **exact-signature construction**, which permanently excludes an intent following its first observable divergence. It is not a consequence of Proposition 1. Thus, the theoretical correction does not require changing the released ground-truth construction, but it substantially narrows the general guarantee that can be claimed for threshold-based CES definitions. We have revised the theoretical interpretation accordingly and no longer attribute general monotonic contraction to Proposition 1 or to ShenJi's learned behavior. We are genuinely grateful for the reviewer's rigorous mathematical scrutiny. This correction has strengthened the theoretical soundness of our work by distinguishing what is mathematically guaranteed, what follows from the benchmark construction, and what remains an empirical property of the model.
AuthorsOpenReview ↗

Response to W2&Q2: Identifiable-Case Performance and Ablation Evidence

We sincerely thank the reviewer for this important and constructive criticism. We fully agree that ShenJi's weak performance on identifiable samples and the limited evidence supporting its individual components raise substantial concerns about its effectiveness. Your observation prompted us to critically reassess the model's capabilities, implementation, and experimental interpretation, leading to significant revisions that improve the scientific rigor of our work.

1. Performance on identifiable samples (Page 9, Lines 452–472).

We acknowledge that ShenJi achieves only 0.187 IC@1 on singleton CES windows, below the uniform chance level of 0.250. As the reviewer correctly points out, these are cases where the observation criterion already distinguishes the intended command, yet the model fails to exploit this information effectively. Our further analysis confirms that the problem lies in misordered compatibility scores rather than the set-decoding threshold. Therefore, the current results do not establish sufficient evidence-aware discrimination capability.

We have explicitly clarified this limitation and now distinguish ambiguity preservation from post-resolution discrimination, rather than attributing the model's failures solely to insufficient observations.

2. Independent ablation evidence (Page 24, Lines 1264–1295, Table 17).

We have clarified that the component ablations involve independent retraining under the same five data-split seeds and two training seeds. Importantly, removing CIGR or evidence memory does not reduce Set-F1; the corresponding values are 0.775 and 0.781, compared with 0.767 for the full model. We therefore no longer claim that these components independently improve set-level performance.

We also re-examined the contraction regularizer (Page 7, Lines 351–367). The reported hard-count penalty provides no gradient, and the earlier pilot ablation used a different implementation. Consequently, we have withdrawn the unsupported claim that this regularizer enforces learned contraction.

3. Revised interpretation and contribution.

We fully recognize that ShenJi remains an imperfect exploratory baseline rather than a demonstrated superior intent-recognition method. Its limitations are now explicitly acknowledged.

More importantly, the central contribution of MIRAGE is its identifiability-aware benchmark, which allows us to distinguish unresolved observational ambiguity from failures of the inference model. ShenJi's poor performance on identifiable samples illustrates why this distinction matters: observational distinguishability does not guarantee that a learned estimator can exploit the available evidence.

We are genuinely grateful for the reviewer's rigorous assessment. It has helped us substantially improve the accuracy of our experimental claims and establish a more appropriate distinction between the methodological contribution of MIRAGE and the current limitations of ShenJi.

AuthorsOpenReview ↗

Response to Q1: Rigorous Clarification of Proposition 1

We sincerely thank the reviewer for this important question and rigorous mathematical scrutiny. We fully agree that the original monotone-contraction guarantee was not justified under the stated assumptions. Your observation prompted us to re-examine the proof, correct the time bound, and reconsider the precise theoretical properties of CES.

First, we have replaced the original monotone-contraction claim with a rigorously derived eventual-exclusion guarantee. Specifically, if d(s)≥δ>ϵod(s)\geq\delta>\epsilon_o for all s≥t0s\geq t_0, then

$ \bar D_{\mathrm{obs},t} =\frac{1}{t}\sum_{s=1}^{t}d(s) \geq\frac{\delta(t-t_0+1)}{t}. $

Therefore, an alternative intent is guaranteed to be excluded for

$ t>\frac{\delta(t_0-1)}{\delta-\epsilon_o}. $

The corrected statement is now Proposition 1 (Eventual exclusion) (Page 4, Lines 205–208), with its derivation provided in Appendix C.4 (Page 20, Lines 1062–1074).

Second, we explicitly withdraw the general monotone-contraction guarantee. A running mean of nonnegative discrepancies need not be monotonic. Consequently, an alternative intent may temporarily re-enter the CES before eventual exclusion. The revised Proposition 1 guarantees eventual exclusion under sustained divergence, not monotonic contraction of the entire CES (Page 4, Lines 209–215).

Third, we clarify that the released MIRAGE ground-truth CES is nested by construction, independently of Proposition 1. Under the exact-signature criterion, an intent is permanently excluded following its first observable divergence. This property is specific to the released construction and should not be interpreted as a general theoretical guarantee for threshold-based CES definitions.

We are genuinely grateful for this correction. It has helped us distinguish the mathematically established eventual-exclusion property from the construction-specific nestedness of MIRAGE, substantially improving the precision and theoretical soundness of our work.

Reviewer_HjpPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies intent recognition under partial observation, in simulated military wargame scenarios. Each episode is assigned a latent commanded intent (ATTACK, HOLD, SEIZE or RECON), which drives its state trajectory, and an observer sees only a partial view (visible units' positions, velocities and relative features). The key difficulty is that different intents can produce very similar observed trajectory prefixes. The paper therefore argues that a model should predict the set of intents compatible with the observations up to time t, and whether that set has already shrunk to a single intent, rather than a single label.

To build ground truth for this task, MIRAGE simulates, for each fixed scenario and random seed, one rollout per intent, and constructs the counterfactual equivalence set (CES) at time t: the intents whose observed prefix lies within a threshold ε_o of the true intent's prefix under an observation distance D_obs (Defs. 1–3). The benchmark defines three tasks: intent-set prediction (predict the CES), identifiability prediction (whether the CES is a single intent), and forecasting when the intent becomes identifiable (predict the time t at which the CES becomes a single intent). The paper also proposes ShenJi, a Transformer+GRU encoder with graph-refined intent embeddings that outputs per-intent compatibility scores and an adaptive threshold, trained so that the predicted intent set shrinks as evidence accumulates.

Critical strengths

S1. The setting is military scenarios innovative and also can help with other setings. The motivation of intent recognition in multi-agent military scenarios as an identifiability problem is also important to intent inference in other settings such as autonomous driving. The proposed benchmark provide labels for which intents remain indistinguishable at each step, which single-label intent datasets cannot provide.

S2. The human study is an interesting justification: under the same partial-observation protocol, both human annotators and ShenJi retain multiple candidate intents, with average prediction-set sizes between two and three.

Critical weaknesses

W1. The introduction frames the problem as general intent inference, but the paper focuses entirely on a simulated military setting that the main text barely describes. Except for the CES definition (Def. 2), many definitions and model archetecture are specific to this setting: the four military commands as the intent space, the hex-grid observation distance D_obs (Def. 1), and the known intent-relation. It is unclear how the benchmark and ShenJi would transfer to other settings such as autonomous driving practically.

W2. The role of counterfactuals in the paper is unclear. The prediction task itself is associational: given an observation prefix o≤to_{\leq t}, predict the intents ZZ behind it (where the value P(Z∣o≤t)P(Z | o_{\leq t}) is big). Counterfactuals enter only in data generation: for a fixed exogenous context uu (scenario, policy and seed), the paper rolls out do(Z=z)do(Z = z) for every intent and builds the CES from the resulting trajectories that look similar.

However, if uu is allowed to vary, two trajectories generated by different intents under different contexts uu and u′u' can also be observationally similar. In that case the other intent is equally compatible with the observation and should belong to a valid CES, yet the construction excludes it because it compares only trajectories that share the same uu. The CES therefore reflects unit-level indistinguishability under a fixed uu rather than what an observer can identify from the observations, and the paper should explain why the counterfactual (fixed-uu) construction, rather than the observational distribution over uu, which is also the right ground truth for this task.

W3. The benchmark is somewhat small (40 scenario–seed groups, 5 test groups, one simulator).

Questions for authors

Q1. How should the results be interpreted? ShenJi does not consistently outperform the baselines.

At α=0.1\alpha=0.1, a baseline (adaptive conformal prediction) achieves higher Set-F1 than ShenJi, although it returns larger sets. Simply returning all four intents achieves an even higher Set-F1 of 0.855. When the methods include the true intent equally often, approximately, ShenJi has higher Set-F1, but adaptive conformal prediction returns smaller sets. Could the authors clarify what would count as better overall performance?

Q2. Several parts of the ShenJi architecture are unclear. ShenJi encodes the observed units over time into an evidence vector hth_t, refines a learned embedding for each intent by message passing over an intent-relation graph (called CIGR), and combines the two into a score per intent. It is not explained what the edges and edge types of the intent graph represent, or how hth_t and the intent embeddings are combined.

6 comments
AuthorsOpenReview ↗

Response to W1: Generalizability Beyond Military Scenarios

We sincerely thank the reviewer for this valuable and constructive suggestion. We fully agree that extending MIRAGE beyond simulated military scenarios would substantially strengthen its scientific contribution and broader applicability. We particularly appreciate this perspective, as it has helped us identify an important direction for improving our work.

Following your feedback, we promptly initiated a focused investigation and preliminary exploratory experiments on potential cross-domain applications. Unfortunately, given the limited revision period and our current domain expertise, we were unable to establish sufficiently rigorous results to support a reliable cross-domain evaluation.

Nevertheless, we believe the core MIRAGE framework is not inherently restricted to military scenarios. Its counterfactual construction, observation-dependent indistinguishability criterion, and CES formulation provide a general methodological foundation. Applying this framework to other domains would require domain-specific intent definitions, observation functions, distance metrics, and adaptations of ShenJi's intent-relation graph. We have explicitly clarified these transferable components and domain-specific limitations in the revised Section 7.1, without claiming experimentally validated generalization.

We are genuinely grateful for this suggestion. Regardless of the review outcome, we intend to continue investigating broader applications and improving the framework's generality and practical relevance.

AuthorsOpenReview ↗

Response to W2: Counterfactual Construction and Identifiability

We sincerely thank the reviewer for this insightful and constructive comment. We fully agree with your observation that different intents under different exogenous contexts may produce indistinguishable observations. This is an important distinction that we had not sufficiently emphasized, and your suggestion has helped us recognize a valuable direction for extending the scientific scope of MIRAGE.

Our current counterfactual construction deliberately holds the exogenous context fixed to isolate the observable effects of changing intent. Its purpose is to establish a controlled benchmark for intent identifiability, rather than to characterize the full uncertainty faced by an observer across varying contexts. We agree that the observational distribution over exogenous contexts captures a broader and practically important notion of identifiability.

In the revised manuscript, we have explicitly clarified this distinction:

  1. Page 4, Lines 177–184: We explain why the counterfactual construction fixes the exogenous context and how this isolates intent-dependent differences.
  2. Page 4, Lines 185–189: We acknowledge that the resulting CES is an instance-level measure rather than a complete characterization of identifiability under the marginal distribution over contexts.
  3. Page 10, Lines 539–539: We explicitly identify the absence of a marginal context model as a current limitation.

We particularly appreciate the broader research perspective offered by this comment. Extending MIRAGE to account for varying and potentially unobserved exogenous contexts could substantially strengthen its practical relevance and scientific value. Although this extension remains beyond our current evaluation, we recognize its importance and intend to investigate it systematically in our future research.

We are genuinely grateful for this suggestion, which has helped us clarify the methodological boundaries of our present work and identify a meaningful direction for further development.

AuthorsOpenReview ↗

Response to W3: Benchmark Scale and Evaluation Robustness

We sincerely thank the reviewer for this valuable observation. We fully agree that increasing the benchmark scale and diversity would strengthen the reliability and broader applicability of MIRAGE. Your comment has encouraged us to further examine the robustness of our current evaluation.

Beyond the reported experiments, we independently regenerated datasets of comparable size across multiple exploratory rounds and observed broadly consistent qualitative patterns. Our formal evaluation also includes five data-split seeds and two training seeds (10 runs) to assess variability across data partitions and model initialization (Page 8, Lines 391–392). These observations provide preliminary evidence of stability under repeated sampling and training, although they do not establish that the current dataset size is sufficient or that the findings are insensitive to scale.

We have explicitly acknowledged the limitations of 40 scenario-seed groups, five test groups per split, and a single simulator in the revised manuscript (Page 10, Lines 534–539).

We particularly appreciate the reviewer's suggestion, which highlights an important opportunity to strengthen MIRAGE. We plan to expand the benchmark to approximately two to five times its current scale, with greater scenario and execution diversity, and systematically investigate how benchmark size affects the reliability of its conclusions. Regardless of the review outcome, we intend to continue this work to improve the benchmark's scientific value and generalizability.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.