acceptodds
Withdrawn from ICLR 2027

IntentTwin: Deconfounding Candidate Support and the Oracle-to-Deployment Gap in Offline Web Behavior Prediction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Offline web-behavior benchmarks frequently insert the demonstrated action into a finite candidate set and then measure reranking. We show that this protocol can create a candidate-support confound: when the target is outside training support while all distractors are sampled from that support, support membership identifies the gold action without modeling behavior. We formalize this effect with support-identifiability rate (SIR), construct support-balanced candidate sets, and evaluate a target-free dual-path proposer that combines support retrieval with open-vocabulary DOM/LM proposals. On filtered OPeRA (4,282/469/992 train/dev/test steps), the original protocol has SIR on target-OOV rows and a support-only rule reaches OOV Exact@1. Support balancing reduces SIR to ; the strongest conditional reranker obtains overall Exact@1, on IV rows, and on OOV rows. The target-free dual-path system reaches Proposal Recall@16 and end-to-end Exact@1 , compared with oracle conditional Exact@1 . On Mind2Web, the same audit yields SIR and an oracle-to-proposal gap of . A 200-row semantic audit labels of exact-key OOV targets as semantically novel, as locator/string variation, and as ambiguous (Cohen's ). These measurements show why support composition must be controlled alongside candidate accuracy when offline results are used to reason about deployment.

Primary area:
reinforcement learning
Keywords:
Offline Evaluation, Web Behavior Prediction, Large Language Models, Candidate Retrieval, Benchmark Auditing, Distribution Shift, Action Generation, Reliable Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.60 over 5 reviews, above 10% of the others.
Reviewer_Gx3TOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper is hard to read, but based on my understanding, it proposes a new subset of the Opera dataset curated for web interactions, a new metric (SIR) for identifying useful datapoints, and focuses on anlayzing mind2web and opera (another web benchmark), and report several metrics that are completely unintelligible. To parse this, here Claude’s summary:

IntentTwin shows that offline web-behavior benchmarks can be gamed through how their candidate sets are built. Benchmarks insert the logged action among 15 distractors drawn from training actions. When the true action never appeared in training (about a third of OPeRA test rows), it is the only unfamiliar candidate, so "pick the unseen one" gets those rows right without modeling behavior. The authors quantify this with a support-identifiability rate (1.0 on those rows) and fix it with balanced candidate sets. Under balancing, a fine-tuned Qwen3-4B reranker still reaches 0.72 accuracy (0.57 on unseen actions). They also measure the gap between oracle reranking and deployment, where the system must propose the action itself. A proposer combining retrieval of known actions with open-vocabulary page and LM proposals reaches 0.52 end-to-end, versus 0.745 with the oracle candidate set. A similar but milder confound and gap appear on Mind2Web. The authors acknowledge that the "IRL-inspired" scorer isn't real inverse RL, that the split holds out sessions rather than users, and that many appendix results are single-seed. Results are also highly sensitive to evaluation context length.

Critical strengths

Graphical visualization of the method in figure 1 and 2

Critical weaknesses
  • The writing is confusing and hard to follow. Right away, it introduces concepts that look simple on surface but simply mean nothing: what is “training support membership”, “support-identifiability rate (SIR)”, “open-vocabulary DOM/LM proposals.”, “target-anchored construction”, etc. Writing should be clear and meaningful. This draft is far from it.
  • Poor description of the data and method. What is Opera? why did you choose it? what training are you performing (since your method is focused on training, as mentioned in abstract and 108)? I needed to look it up, which should have been properly explained in the paper.
  • This work does not meaningfully advance the state of web agent research. It does not focus on more benchmarks web benchmarks beyond Mind2Web, which was released in 2023. There are a lot of high quality benchmarks that came out even in the past year (e.g. WebChain)
  • It lacks substantial baselines, like recent LLM-based approaches like GenericAgent from BrowserGym (which can be powered by open-weight models like Qwen3.8 32B). To show that their contribution is useful, they need substantial baselines.
  • Significant concepts missing. It does not discuss top-k-based approach like Dense Markup Ranker from WebLINX or the accessibility-tree based approach from WorkArena.
  • Narrow contribution: the authors describe their work as “train-only candidate contract, row-level likelihood ledger, and explicit target-OOV analysis for a closed-choice factorized scorer.” This sentence does not make much sense, but from what i could gather, it appears that this approach is only useful during the training process, yet they do not release a trained model.
Questions for authors

It feels the paper is primarily written by Claude. Is that the case? It is okay if it’s for a draft, but I expect the final paper to be properly rewritten to be clear and direct about the contributions. Otherwise, it's not a good use of reviewers’ time if the paper is written in a way that's hard to parse and not informative.

Reviewer_hFojOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces support-identifiability rate (SIR) to quantify this shortcut, constructs support-balanced candidate sets, and separates proposal coverage, ranking accuracy conditional on coverage, and end-to-end selection. It also proposes a context-gated dual-path candidate generator combining training-support retrieval with open-vocabulary DOM/LM proposals, supported by reproducible candidate-level records.

On filtered OPeRA, with 4,282/469/992 train/dev/test steps, a support-only rule achieves 100% accuracy on unseen-target rows under the original protocol. Balancing candidates eliminates unique-support identification; the QLoRA reranker retains 72.14% overall accuracy and 56.68% unseen-target accuracy. Without inserting the target, the dual-path system achieves 69.15% Recall@16 and 52.12% end-to-end accuracy, versus 74.50% oracle-candidate accuracy—a 22.38-point gap. Proposal ablations and candidate-composition sweeps provide additional checks.

Broader evidence and scope: Mind2Web shows analogous oracle-to-pipeline gaps of approximately 20–23 points across its three splits. A two-annotator audit of 200 exact-key unseen targets finds 71.5% semantically novel, with 24.5% reflecting locator/string variation. These are offline action-selection results, not online browser-task success; OPeRA’s primary split also permits recurring users.

Critical strengths

The paper identifies a concrete evaluation shortcut and demonstrates its severity directly. When the gold action is the only candidate absent from training support, a context-free support rule achieves 100% accuracy on OOV rows. The SIR diagnostic and support-balanced protocol turn this observation into an actionable benchmark audit. This is a substantive strength because it exposes a mechanism that can invalidate interpretations of apparent behavioral generalization.

It empirically separates candidate discovery from conditional ranking. On OPeRA, removing the oracle target anchor reduces accuracy from 74.50% to 52.12%, and Mind2Web exhibits similar gaps across three generalization splits. The proposal baselines and ablations help substantiate this distinction. This supports acceptance because the evaluation provides a more informative account of practical action-prediction performance than oracle-candidate accuracy alone.

Critical weaknesses

The central proposed method is insufficiently specified. Appendix A.2 describes the dual-path proposer at a schematic level, but does not explain how the context gate is trained, how DOM/LM proposals are generated and constrained, or where the out-of-support distractors for balanced evaluation originate. These choices directly determine candidate coverage and difficulty. Their omission makes the headline gains difficult to reproduce or assess independently, weakening the paper’s main empirical contribution.

Eliminating SIR does not establish that the evaluation isolates behavioral understanding. SIR detects only whether the gold action has a unique support status; zero SIR does not rule out other support-related or candidate-construction shortcuts. Indeed, the balanced support-only baseline still achieves 25.38% OOV accuracy. Without context-removal/shuffling controls or stronger candidate-only baselines, the retained reranking accuracy does not establish context-dependent discrimination as claimed.

Questions for authors

How is the context gate trained, how are DOM/LM proposals generated and constrained, and how are out-of-support distractors sourced for balanced evaluation? Please clarify which data each component accesses and whether these details are documented in the supplement.

What evidence establishes that the balanced reranker uses behavioral context beyond candidate-only cues? Are context-removal/shuffling controls or stronger candidate-only baseline results available? note that SIR = 0 excludes only unique-support identification.

Reviewer_9FjEOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper figures out the candidate-support confound problem in offline web-behavior benchmarks and proposes a support-balanced candidate protocol to resolve this problem.

Critical strengths

This paper has defined a critical training information leakage problem in this area, which affects training and prediction accuracy in the web-action prediction tasks.

Critical weaknesses
  1. The paper needs more details in the background. Many terms and concepts are not explained, like what does a distractor look like, what is an observation in web traces, what does "support" mean in "support-only" and "out-of-support", etc. This makes the paper not self-contained enough.

  2. The paper does not clearly connect its stated research problem, methodological components, and experimental evidence. For example: (1) Figure 2 proposed a multi-loop method for factorized decision path, but many modules in this figure is not explicitly explained in the methodology or in appendix. For example, the "PathFaith", "Factor Masking" and "Factor Permutation" in module 8 are never introduced in the paper.

Questions for authors

Could the authors explicitly map each contribution to the component that implements it and the experiment that evaluates it?

Reviewer_R3PmOpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper audits the evaluation protocol of offline web-behavior prediction benchmarks (filtered OPeRA and Mind2Web). The authors observe that the standard "target-anchored" construction inserts the gold action into a fixed 16-way candidate set while drawing all distractors from a frozen training vocabulary. Consequently, for any test row whose gold action is out-of-vocabulary, the gold candidate is the unique out-of-support option — so a trivial "support-only" rule that never reads the context achieves OOV Exact@1 = 1.0. The authors call this candidate-support confounding.

To address it, this paper contributes: a diagnostic, the support-identifiability rate (SIR), measuring how often support status alone uniquely reveals the gold, and introduce a support-balanced candidate protocol to eliminate it. Additionally, they propose a target-free dual-path framework (combining support retrieval with open DOM/LM generation) to quantify the oracle-to-deployment gap.

Critical strengths
  • The proposed SIR + support balancing approach seems to be technically sound.
  • This paper makes an acute, original observation regarding the candidate-support confound, and proposed approach is evaluated wth couple of datasets.
Critical weaknesses
  • limited novelty and significance. Experiments are only conducted with two datasets, I am mot quite wether this is a common issue in such or similar tasks. More importantly, the authors did not provide sufficient information about the significance of the proposed approach.
Questions for authors

See the above

Reviewer_rvV9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Some web-agent benchmarks ask a model to pick the user’s next action from a list of options. If the wrong options all come from actions seen in training, a new correct action is the only unfamiliar option, so a model can pick it without understanding anything.

The paper names this flaw and gives a way to measure it. It fixes the flaw by mixing several unfamiliar options into every list. It also tests a more realistic setup where the system has to come up with its own options.

On the OPeRA shopping dataset, picking the unfamiliar option gets every new-action case right before the fix and does no better than chance after it. A real model still gets about 72% right after the fix. When the model has to come up with its own options, accuracy drops from 75% to 52%. Mind2Web shows a smaller version of the same flaw and a similar drop of about 20 points.

Critical strengths
  1. It finds a concrete, checkable flaw in how these benchmarks are scored, and gives a cheap fix. Picking the only unfamiliar option, without reading the context, gets every new-action case right. That makes it clear that high accuracy under this setup can come from the candidate list rather than from understanding user behavior. This matters because a scoring flaw affects every method evaluated this way, and the fix (mixing unfamiliar options into every list) is easy to adopt.
  2. It measures how much the easy test setting overstates real performance. By making the system come up with its own options instead of being handed a list with the answer in it, the paper shows accuracy dropping by about 20 points on both OPeRA and Mind2Web. This matters because it tells readers how far offline results are from what a deployed system would achieve.
Critical weaknesses

The shortcut is guaranteed by the way the authors build their own candidate lists, but the paper does not show that any published benchmark or result is affected. Without this, it is unclear how much the finding matters in practice. Re-scoring a published method or result under the balanced protocol would show whether this flaw changes any existing conclusions.

Several numbers contradict each other or look like pipeline bugs (the Table 18 NLL, the Mind2Web results in Table 5, the collapse at 1,024 tokens), and most diagnostics use a single seed on a small test set.

SIR reaching zero only removes one cue, and the balanced protocol still has a similar action-type shortcut, with no candidate-only baseline to check what other cues remain.

Questions for authors

Why does accuracy fall to near chance (0.097) at 1,024 tokens, when it is 0.43 at 768 and 0.745 at 2,048? Could this be a bug in the evaluation pipeline, such as the candidate list being cut off, and does the same issue affect the 384-token results?

Which truncation side is used for the main results, and how many rows are truncated at each prompt limit? Is the candidate list or the scored completion ever cut off?

The persona and segment arms add tokens to the prompt, so under the 384-token limit they may lose more history or candidates than the baseline arm. Can the authors rule out that their negative results come from truncation rather than from the added information?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.